The Truth Finds A Way

One trend that continues to gain momentum in the BI world is self service business intelligence, and it has IT groups concerned that the whole focus of a BI team is to champion a single version of the truth. MicroStrategy just released it’s desktop application for free, and version 10.6 is now available. If you have not taken a look at it, it is worth spending some time on. Tableau has built virtually it’s entire business model on self service BI. Any department that can’t get enough resources from the BI team can go and build their own dashboards now. Qlik is the same way. Now Microsoft’s Power BI has stepped into the ring with a growing offering. Alteryx, Sisense, Birst, Zoho – all of these are pouring resources into self service BI. IT groups are running scared, and maybe rightly so. Self service BI doesn’t have to have any training, any experience, any skill sets, any data governance, any single vision of what the single version of the truth is supposed to look like. Power to the people. Democratizing data like never before. Gartner is even saying that this is going to be the death of BI.


Or is it? People have always gone outside of the BI or the IT teams to build reports and analysis. It’s called Excel, or Access, and its been around for, oh I don’t know, a few decades? I’ve seen entire departments run from Excel and Access applications. You can’t stop them from using these. I’ve seen desktop computers that used to belong to an enterprising employee 10 years ago, that built an Access application that became mission critical to a department, and the department has seen the employees turnover two or three times in that period. Nobody remembers who the original developers were, or even what it was running, but each newly appointed department head got the instructions to make sure that desktop computer remained powered on under their desk and hooked up to the network. God help them is this computer dies or something. And these scenarios are a nightmare for IT groups that get handed to them to support – but you cannot stop it. Much like that Jeff Goldblum line from Jurassic Park – Life finds a way.


jp93-eggs1These new tools just give the enterprising users a new means to create things that BI teams or IT groups are going to have to support. They are given tasks to run the business, and then the BI group doesn’t have the time or means to provide them the reports or analysis they need to meet those new milestones that leadership keeps placing on them. And, nobody likes a whiner, so they invent what they need outside of the process. Business is happy. Users are happy. IT is blissfully unaware. All they know is that they stopped emailing them asking for a status on their request and didn’t even miss the emails. Is this really such a bad thing? I mean, if the BI team built every single thing the business thought it needs to run then they would collapse under their own weight. Rather than cringing at these outlaw scenarios, if you look at them as a proof of concept exercise, and let the POCs that live life past a year or some determined amount of time that proves the reasoning and the needs were real, then really everyone who does these are helping out the BI team.


Rather than seeing these activities as competition or amorphous growth that cannot be supported, BI teams should looking to guiding these rogues in a way that helps keep some sanity to possibly taking over the project when it has matured. Choosing one self service BI tool and embracing it, training on it, training others on it, would be a much better alternative than leaving it up to the department and needing expertise on 5 or 6 different tools de jour. For one, it enables you to hire or train a resource for the BI team to assist users in developing their own projects and for taking over projects that have grown to big to be a department only project. Secondly, it provides some consistency to the rogue POCs so that the company looks like it knows what it is doing. Third, taking the lead to facilitate this movement means BI and IT can guide the company down the path to some degree, rather than being handed who knows what to support. Don’t fight the tidal wave. Grab it, embrace it, lead it. Sticking your head in the sand is never a good strategy.


#CPG: 10 Reasons Why Now Is the Time to Get into #BigData

Rick Delgado is a new technologies freelance writer. In this article he takes an objective view of Big Data and lays out 10 reasons why now is the time for Big Data. My observations are italicized below.

1. It will keep your data secure

Being able to completely map your data out in front of you will make you better at analyzing potential threats. You can easily ensure that the most sensitive data you have in your system has all of the necessary security features it needs to stay well protected and within any sort of regulations that your business has to adhere to.
Many BI tools have provided for this sitting on top of the data – however – the big data structure allows this to be mapped into the clusters where data lives.

2. It opens up brand new revenue sources

You will gather a ton of insight into your part of the market. This insight is valuable, and not just to you. Selling out non-personalized industry data about trends in the market can bring in a ton of money from any other businesses operating in the same portion of the industry that you are, making it a great way to bring in some more funds.
While this is not always possible for a CPG company, for example, to do when retailer-sourced data is in the mix there are other opportunities at the enterprise level.

3. It’ll give you an advantage

Businesses are built on tradition. Any industry is going to have years of tradition that believe that one way of doing things is the right way and if it isn’t broke, don’t fix it. One important value that big data has is that it lets you really examine and analyze preconceived notions about aspects of a business you may have never even considered. It will provide better data when it comes to experimenting and trying to innovate, meaning that you’ll be able to get a much better advantage over your competitors.

4. Better Visuals

Big data requires cutting edge data visualizing tools in order to translate all of those numbers and data points into something a bit more tangible. This will increase overall usability that can be utilized by people in the business themselves or their end users.
There are increasing numbers of tools available for the user to create insight from their data lake – Vortisieze prefers MicroStrategy but ultimately is agnostic toward the BI tool selected giving you maximum flexibility.

5. It’s very easy to set up

Big data used to be something that only big corporations could afford to get into. Luckily, there are alternatives that can bring the benefits of big data to a company of any size. Hadoop is a great open source framework that can handle large amounts of data. Hadoop works with Excel, which most businesses already use.
The entry point is such, especially with the ABC of business (Analytics, Big Data, Cloud Computing), that retailer sales/support offices can easily add this as a budget item without involving their IT division.

6. It helps your business evolve

One of the best things about big data is just how scalable it is. Getting the right tools means being able to convert not just number data, but also text, audio, and video files to find patterns that can give a business a ton of insight into the market. Investing in big data as a small company means you’ll be able to easily handle all the data easily that comes with growth.
You begin seeing your business as your customers (the ones buying your products from the retailer) see your business.

7. It can significantly cut back on maintenance

Businesses that rely on large pieces of equipment often spend tons of money when it comes to replacing and maintaining them using predetermined data that tries to predict how long they’re going to remain efficient. Big data allows you to get rid of rounded averages and replace them with specifics, allowing you to squeeze even more life out of machinery that still has plenty of use left.
And with cloud computing, also called data-as-a-service, there is no equipment involved to maintain.

8. It brings you closer with your customers

The customers of today are a lot different than in the past. The rise of the internet allows them to thoroughly and tirelessly research a product before buying and to communicate with tons of people about which brand they should do business with. Big data allows you to better profile these fickle consumers and figure out specifically what they want.
See comments to #6 above.

9. It’s easier to analyze risk

There’s more to business than running your own company effectively. Every business is just one in a large industry that must regularly compete and innovate in order to stay ahead of the rest. Utilizing big data to analyze things like news articles and social media will give you cutting edge information about the biggest and latest developments in your industry.

10. You’ll be able to improve your products

Big data is a key that’ll explain specifically what your consumers really think about your product. Big data is even being used in medical research for companies that do personalized medicine or companion diagnostics, and need to analyze large amounts of biological data. You’ll be able to use your insight to easily get a better picture of your customers based out of different geographic areas and belonging to different demographic groups. Once you’ve seen how the product is perceived, you can easily raise efficiencies in key areas along the product process.
These are extremely valuable insights to present to your retailer buyer – enabling your company to rise above the noise from your competition.


Source:  10 Reasons Why Now Is the Time to Get into Big Data





#BigData – All of the Hype – None of the Calories!

There has been a lot of hype in the IT industry around Big Data over the last year. It’s been quite breathtaking watching the polarization that has come with this hype, and I have found that everyone falls into one of three camps. When you are reading or listening to some rhetoric about big data, it helps me to classify the argument into one of these three camps, that way I can discern what I am hearing or reading more quickly.

Camp #1 – They are selling something else. So, company A has a mobile app that collects data from a MSSQL server in the cloud, handles maybe 500 GB of data, and they don’t really have a clue what big data is all about. But, everyone is talking about big data, so they want to get into the game. They use phrases like “We’ve always been about big data” or  “We’ve been doing big data for 20 years”, or “We go from big data to smart data”. I’m sure they have a great mobile app, or they have a really clever data model in Teradata – but I generally see these as red flags as NoBigData. No big deal. Maybe the convolution they are creating around big data is generating some big sales for them. I don’t know. Big data sounds cool, right? Surely everyone does big data? If you find the conversation about big data quickly devolves to the cool user interface, or real time data collection via a mobile app, chances are they are just selling something else. I don’t have a problem with non big data products. Not everything should be a big data project to be completely honest. Just don’t call your product big data when it isn’t. It’s ok, really. My car or my DVD player aren’t big data either, but I use them regularly.

Camp #2 – They’ve been ordered to implement big data. These are the teams that know they have a mobile cloud app or a clever DB2 data warehouse, but the CIO has given the mandate that” everyone is doing big data, and so shall we all”. These folks would rather have their arm caught in a wood chipper than feed the big data hype, and they will do everything in their power to kill the initiative without sounding like an obstruction. Generally, I see these folks as hardened IT folks who have made their careers on another database platform, are comfortable with it, heck may even love it. It’s like asking them to cheat on their significant others with someone they don’t like. If it ain’t broke, why replace it? You see these teams, trying to install Hadoop on a TI-85 calculator, and then loading 5 TBs of data, and then complaining the query took an hour, so big data is busted. Maybe. Probably not. I bet they didn’t read the manual, because, you know – that’s against the code….wink wink.

Camp #3 – They get it. To be fair, some of these concepts require a completely new understanding on how data works. Schema on read, for instance. This just boggles the traditional RDBMS developers mind. Typically, you look at your data, create a table, set it’s column data types (these columns are integers, these other columns are strings), then load the data, and then output it to a report. This is called schema on write. I define the table, when I write the table structure. With Hadoop, We load the data, then figure out which columns are integers or strings, and then output the data to a report. This is called schema on read. Why is this significant? I can change the entire data model in 5 minutes. That is not very easy in Oracle or MSSQL. If I built an entire reporting application, and realized a year down the road that what I thought was an integer was supposed to be a character, I can still fix that in 5 minutes on a Hadoop platform. It would take longer than that to put together a project plan to do that in Oracle. I can also add a column to the end of my table, and then figure out what to do with it 6 months later. These concepts are so far removed from traditional databases that they just do not compute. But, when you see someone truly understand what these new super powers bring, then you can see the excitement. Speed, scalability, agility, cost savings – every single one of those are worth a consideration. Maybe when you put all four together in the same sentence, people just assume they are too good to be true?




#MicroStrategy 10 Adds New #Hadoop #Analytics Capabilities

The leader in the BI analytics platform sphere, and the preferred BI tool by Vortisieze, MicroStrategy is now allowing users to enjoy the richness of data that Hadoop allows.

  • MicroStrategy Inc.’s has launched a major upgrade to its namesake business intelligence platform that promises to help organizations make more out of the vast troves of data they’re storing in Hadoop. That’s done through improved connector bundled natively into the new release.
  • MicroStrategy 10 enables analysts to manipulate imported information using new search and visualization functionality that the company says removes the need to use external tools for that job. That avoids the overhead associated with moving files back and forth, which can amount to significant time savings at the petabyte scale in which many production-grade Hadoop clusters operate.
  • The company promises to take even more delay out of the equation with optimizations to its multi-dimensional analytic algorithms, which now makes it possible to put more data into the logical constructs used to group similar metrics for faster queries. MicroStrategy said that the update has helped PayPal, Inc. achieve sub-second response times for 400-gigabyte workloads.
  • Analysts can take advantage of that performance through the improved interface also introduced in the new release, which is extended to mobile devices with a new companion app that can track the the performance of each user. The client also includes a badge-based security system that allows administrators to control who can access what and how.

Source:  MicroStrategy 10 brings new analytic capabilities to the Hadoop table





Key Action Steps #CPG Companies Should Do With Trends in #BigData

  1. Value a different set of skills – Beyond specific roles, people with the ability to interpret data and make it actionable — who can answer the “why” to big data’s “what” — will be of particular value to employers. They also can’t be replaced by robots.
  2. Unlock new markets – Identify the opportunities that were previously closed due to a lack of information. Airbnb’s example is telling, since the startup was able to identify a home rental market that had long been stagnant due to a lack of trust, which in turn was the product of scarce information on the product being sold, and find value in spare assets.
  3. Steward your data – Companies should be aware of their role as trustees of data and identify risks in their data assets. Customers have entrusted their data to those they engage with their business. Absent the abuse of user data or clear harms, there can and should be a large scope for consumer-friendly innovation.
  4. Make small data a competitive advantage – Just as the Internet of Things breaks connectivity down to its constituent parts, small datasets from smart devices with specific attributes will become increasingly valuable. Democratized information increasingly demands individualized service that large, more centralized datasets are ill-suited to provide.

Source:  A Business Guide to Going Big With Big Data




Five Ways #BigData is Shaping #CPG Business Right Now

  1. Democratizing information – Information used to be a precious good; now it’s a cheap commodity, and that’s a good thing for consumers and the companies servicing them. Accessibility to data is empowering. The acquisition and spread of knowledge no longer depends on centralized gatekeepers, but on distributed providers.
  2. Highlighting the customer – Companies can now know what customers think and want — and how to fulfill that. Larger, cheaper datasets and the tools to analyze them are paying dividends. Consumers in turn are living quantified lives.
  3. Growing assets – Balance sheets will soon have a new line for data stores, while identifying new uses for once latent assets, such as property. Every company is a data company, whether it knows it or not.
  4. Overthrowing hardware – Data-driven business models are infinitely scalable through software, with higher margins, and can be delivered online and through connected devices. This is one reason why cars are simply smartphones on wheels.
  5. Lowering startup and operational costs – Existing companies can do more with less, while lowering the barriers to new entrants.

Source:  A Business Guide to Going Big With Big Data




#BigData Will Power #CompetitiveAdvantage for #CPG Business

87% of enterprises believe that big Data analytics will redefine the competitive landscape of their industries within the next three years

The combination of social networking as a way of life, and the ever-growing number of devices on which people stay connected, continues to generate untapped sources of data that could help businesses compete more effectively.

Decreasing cost of both storage and computing power have made it feasible to collect this data – which would have been thrown away a few years ago. Today, enterprises want to capture and manage large amounts of information from multiple sources both structured such as ERP, CRM and database systems and unstructured including web sites, social media and streaming.

“More than ever, growing adoption in mobility, new types of applications and cloud services are driving organizations and consumers to store, manage, safeguard, analyze both traditional, structured data as well as unstructured or semi-structured data generated by multiple applications running on smart phones, tablets and all other computers. Whether it is called Big Data or Internet of Things, we all want to gain more insights on how to run our business more effectively. Technology companies such as Microsoft hold the belief that it is more crucial to embrace advanced analytics that will enable everyone in organizations to gain actionable insights and gain competitive advantage by benefiting from predictive and prescriptive analytics,” says Aydin Gencler, Director Product Marketing, Middle East & Africa at Microsoft.

Big Data main tool Hadoop was driven by the exponential growth in the volume of unstructured data from Big Data analysis and the ability to access data at high speed with reduced costs compared to traditional RDBMSs.

A brighter future for Big Data


Analytics, big data and cloud computing will continue to transform the business landscape for the foreseeable future.




 #Vortisieze Technology Releases First Software Product for #CPGMarketing

For Immediate Release

Vortisieze Technology LLC

Lowell, AR – March, 3/9/2015 2015 – Vortisieze Technology has released their first software application, Filter Manager. This application automatically updates filters built in the MicroStrategy business intelligence tool based on a calendar table, and is completely customizable and extensible. For analytics that do not follow a standard calendar format (like a Walmart fiscal calendar for instance where the year beginning date can change based on an odd number of weeks every four years), it can be difficult to build rolling date filters for scorecarding metrics, or in some cases downright impossible. This application turns that complex logic into simple date filters that are then easy to use in your reports and metrics.
This software rounds out their application stack that provides item-store-day level analytics built on top of Cloudera Hadoop and MicroStrategy. By combining these tools with best of breed analytics built in MicroStrategy and tools like Filter Manager, Vortisieze has created a fast, agile data platform that can react to your changing business as fast as you need it to. By Partnering with Cloudera as their Hadoop provider they are able to scale to any size of dataset that is needed. Data that was just a dream is now possible, from correlating sales performance to social data and weather data, to finding relationships between SKU-Store performance and hospital mortality or unemployment rates, Vortisieze is pushing the boundaries of thinking that can be used to increase sales.

Vortiseze Technology LLC builds analytics platforms in MicroStrategy and Cloudera-Hadoop. Filter Manager is their first go-to-market application that can be purchased on it’s own to use with your own data warehouse or in conjunction with the Vortisieze MicroStrategy-Cloudera application that helps CPG companies manage their business with very large retailers.

Vortisieze Technology LLC is a software and data analytics company with over 30 years total experience in data warehousing, MicroStrategy solutions, software development, Walmart category management reporting and Cloudera implementations. Based in Northwest Arkansas they are centrally located to service their customers who deal with very large retailers.

If you are interested in Filter Manager or our complete analytics suite please email or call us at 479-633-7821


# # #





25 Surreal Facts about #BigData You Should Know

There have been a lot of negative connotations about the term “Big Data”. I watch DBAs cringe when you mention it. Whether or not you like the term, the fact of the matter is that we are now generating more data in the world and at an explosive rate – faster than we have ever seen. Take a look at this list of facts about how much and how fast we are generating data today.

  1. Over 90% of the world’s data was created in the past 2 years. [Source]
  2. In 2 days we generate as much data today as what was generated in our entire history up until 2003 [Source]
  3. The total volume of data being generated by industry doubles every 1.2 years [Source]
  4. Experts think that by 2020 the total amount of data in the world will grow from 3.2 zettabytes today to 40 zettabytes. [Source]
  5. Retailers could increase their profit margins by more than 60% through the full exploitation of big data analytics. [Source]
  6. Every 60 seconds we fire off 204 million emails, click 1,8 million Facebook likes, Tweet 278 thousand times, and post 200 thousand photos to Facebook [Source]
  7. About 100 hours of videos are posted to YouTube every minute and it would take 15 years to watch every video uploaded one day. [Source]
  8. Google processes an average of 40 thousand searches per second, over 3.5 billion in a single day. [Source]
  9. If you burned all of the data created in one day onto DVDs, you could stack them up and reach the moon – and back. [Source]
  10. It is believed that AT&T has the world’s largest database – its phone records database is 312 terabytes in size, and almost 2 trillion rows. [Source]
  11. 1.9 million Big Data jobs will be created in the US by 2015, and those will be supported by 3 new jobs created outside of IT – totaling of 6 million new jobs. [Source]
  12. Data transferred over mobile networks increased by 81% to 1.5 exabytes (1.5 billion gigabytes) per month from 2012 to 2014. Video makes up 53% of that total. [Source]
  13. The NSA is believed to analyze 1.6% of all global internet traffic – 30 petabytes (30 million gigabytes) daily. [Source]
  14. The Hadoop market is expected to grow from $2 billion in 2013 to $50 billion by 2020. [Source]
  15. Facebook users share 30 billion posts every day. [Source]
  16. 570 new websites are created minute. [Source]
  17. The number of electronic bits of data in existence is believed to have exceeded the number of stars in the physical universe in 2007. [Source]
  18. This year, there will be over 1.2 billion smart phones in the world which are constantly collecting data from sensors, and the growth is predicted to continue. [Source]
  19. The Internet of Things will mean that the number of devices that connect to the Internet will rise from about 13 billion today to 50 billion by 2020. [Source]
  20. 12 million RFID tags had been sold in 2011. By 2021, it is estimated that number will rise to 209 billion as the Internet of Things takes off. [Source]
  21. Data centers now occupy an area of land equal in size to almost 6,000 football fields. [Source]
  22. Almost as creepy as the movie Minority Report – Big data has been used to predict crimes before they occur– a “predictive policing” trial in California was able to identify areas where crime will occur three times more accurately than existing methods of forecasting. [Source]
  23. By better integrating big data analytics into healthcare, the industry could save $300 billion a year, according to a recent report – that’s the equivalent of reducing the healthcare costs of every man, woman and child by $1,000 a year. [Source]
  24. The big data industry is expected to grow from US$10.2 billion in 2013 to about US$54.3 billion by 2017. [Source]

So what is your Big Data strategy for your company? Still don’t think you need one?


What’s next for #CPG #CategoryManagement?

What do Walmart, Facebook, Yahoo, Twitter, IBM, Google, EBay, Teradata, LinkedIn, Hulu, The New York Times, MicroStrategy, and P&G have in common? They are all harnessing the power of Hadoop to store, serve, slice, and rationalize Big Data to advance their business. To peer into volumes of data like never before. Volumes of data that have been too big to be this nimble and uncovering things about their customers that they never dreamed possible – until today.

The biggest downside to standard data warehousing and BI tools today is that you have to know the questions you want to ask ahead of time. This creates a never ending search for patterns, outliers, and relationships in your data. If you dream up a question your existing architecture doesn’t support, you have to involve IT or software vendors and re-architect the whole data warehouse.

What if you could gaze into a magic 8-ball and it would tell you everything you needed to know about your retail category – all of the SKU changes to maximize sales,  your out-of-stocks and phantom inventory, your sales by geography or store traits, plus patterns in your data that you did not even know to look for. Welcome to the next generation of BI data warehousing in retail category management – Hadoop!

Why Hadoop?

  • Hadoop is powering today’s Big Data initiatives and is gaining more and more acceptance across many different business units. Coupled with Hive, Pig, Scoop, MapReduce, and numerous others, there are multiple robust ways to attack and slice your data.
  • Your original data formats are unchanged, so you can reuse them in their raw form at a later date. This guarantees no data loss in case you think of some way to explore your data in the future that you have not thought of today. It also does not lock you into a proprietary third party data format.
  • No ETL is required. Data is loaded into the HDFS and then you are done. Then use coupled tools to go unearth the data you are looking for rather than churning it into a cookie cutter format that you hope will give you insights.
  • Hadoop is scalable using inexpensive hardware. Add nodes to your cluster all day long, using junker PCs you have lying around in the closet. No longer do you need a $50K RAID SAN to house and protect your data. Running out of space after 5 years of category data? Just load up some more nodes and you will be good for another few years.
  • Hadoop couples with several analytics vendors – MicroStrategy, Pentaho, Zoomdata, SSRS, Tableau, SAS, with other open source products as well as numerous several built-in packages.

We are breaking new ground focusing on implementing Retaillink or other Demand Signal data in a Hadoop cluster, and applying several analytics packages on top of that to let this new Big Data platform shine in the category management space like never before.