Skip navigation
Help

cloud

warning: Creating default object from empty value in /var/www/vhosts/sayforward.com/subdomains/recorder/httpdocs/modules/taxonomy/taxonomy.pages.inc on line 33.
Original author: 
Todd Hoff

This is a guest post by Yelp's Jim Blomo. Jim manages a growing data mining team that uses Hadoop, mrjob, and oddjob to process TBs of data. Before Yelp, he built infrastructure for startups and Amazon. Check out his upcoming talk at OSCON 2013 on Building a Cloud Culture at Yelp.

In Q1 2013, Yelp had 102 million unique visitors (source: Google Analytics) including approximately 10 million unique mobile devices using the Yelp app on a monthly average basis. Yelpers have written more than 39 million rich, local reviews, making Yelp the leading local guide on everything from boutiques and mechanics to restaurants and dentists. With respect to data, one of the most unique things about Yelp is the variety of data: reviews, user profiles, business descriptions, menus, check-ins, food photos... the list goes on.  We have many ways to deal data, but today I’ll focus on how we handle offline data processing and analytics.

In late 2009, Yelp investigated using Amazon’s Elastic MapReduce (EMR) as an alternative to an in-house cluster built from spare computers.  By mid 2010, we had moved production processing completely to EMR and turned off our Hadoop cluster.  Today we run over 500 jobs a day, from integration tests to advertising metrics.  We’ve learned a few lessons along the way that can hopefully benefit you as well.

Job Flow Pooling

0
Your rating: None

The inside of Equinix's co-location facility in San Jose—the home of CloudFlare's primary data center.

Photo: Peter McCollough/Wired.com

On August 22, CloudFlare, a content delivery network, turned on a brand new data center in Seoul, Korea—the last of ten new facilities started across four continents in a span of thirty days. The Seoul data center brought CloudFlare's number of data centers up to 23, nearly doubling the company's global reach—a significant feat in itself for a company of just 32 employees.

But there was something else relatively significant about the Seoul data center and the other 9 facilities set up this summer: despite the fact that the company owned every router and every server in their racks, and each had been configured with great care to handle the demands of CloudFlare's CDN and security services, no one from CloudFlare had ever set foot in them. All that came from CloudFlare directly was a six-page manual instructing facility managers and local suppliers on how to rack and plug in the boxes shipped to them.

"We have nobody stationed in Stockholm or Seoul or Sydney, or a lot of the places that we put these new data centers," CloudFlare CEO Matthew Prince told Ars. "In fact, no CloudFlare employees have stepped foot in half of the facilities where we've launched." The totally remote-controlled data center approach used by the company is one of the reasons that CloudFlare can afford to provide its services for free to most of its customers—and still make a 75 percent profit margin.

Read 24 remaining paragraphs | Comments

0
Your rating: None

The centuries-old scientific and engineering idea of progress through observing, modeling, testing and modifying is under attack. Now it is better to collect and examine lots of data, looking for patterns, and follow up on the most promising. The latest example: Autodesk says designers should generate a thousand product versions.

0
Your rating: None

Clouds

I get press releases every week about some new (or old!) company and their so-called cloud solution. Some folks are clearly abusing the popularity of the “cloud” buzzword, and others are actually doing interesting things with distributed computing, infrastructure- and platform-as-a-service, orchestration, and related technologies. Amazon is the prime mover on IaaS, but OpenStack, CloudStack and Eucalyptus are all making strong plays in that space. VMware’s Cloud Foundry and Red Hat’s OpenShift are pushing open source PaaS, while services like Heroku, Engine Yard and dotCloud (among others) are pushing to be your hosted PaaS solution.

It’s not surprising that so many people are looking to differentiate their cloud solutions, and on the balance I think competition is a good thing that eventually benefits end-users. But as things stand today, it strikes me as exceedingly hard to formulate a comprehensive “cloud strategy” given the plethora of options.

If you care strongly about open source, that helps limit your options. VMware’s Cloud Foundry has been open source for quite some time, and recently celebrated its first birthday. Red Hat’s OpenShift is not yet open source, but work is underway to remedy that. Red Hat, obviously, has a long history of successfully open sourcing their work. Red Hat also recently announced that they would be a platinum member of the newly reorganized OpenStack governing board. VMware, on the other hand, is not a company with which I readily associate open source culture or success; and I don’t see a very robust ecosystem coalescing around Cloud Foundry. Hopefully that situation improves.

And there’s also Canonical, the folks behind the Ubuntu Linux distribution. Canonical has made a real effort to advocate for OpenStack, but their actual contributions to OpenStack don’t seem to tell the same story. Rather than focus on directly contributing to IaaS or PaaS offerings, Canonical is busy making helper products like Metal-as-a-Service and their newly announced “Any Web Service over Me” (with the righteous acronym AWESOME) which aims to provide an API abstraction layer to facilitate running workloads on Amazon’s cloud and on an OpenStack cloud.

The end result of all of this a lot of ambiguity for customers and companies looking to deploy cloud solutions. If you want a private cloud, it doesn’t seem to me that you can make a decision without first reaching a decision as to whether or not you will eventually need to use public cloud resources. If so, your choice of private cloud technology demonstrably hangs on the long-term viability of your intended public cloud target. If you think Amazon is where it’s at for public cloud, then it seems that Eucalyptus is what you build your private cloud on (unless you want to fiddle with even more technology and implement Canonical’s AWESOME). If you think Rackspace is where it’s at, then OpenStack is a more appealing choice for you. But what if you’re wrong about your choice of public cloud provider?

As such, I’m curious to learn what you — the reader — are currently doing. Have you made a technology decision? Did you go all in, or are you leaving room to shift to a different provider if need be? Did you go IaaS or PaaS? Are you a new company, or are you an established organization moving existing workloads to new platforms? Finally, I’m particularly interested to hear from folks in regulated industries — banking, health care, insurance, etc — where your decision as to where to run your applications may be predicated on legal issues.

0
Your rating: None

Signs are emerging that Google is de-emphasizing its efforts in online productivity tools that compete with Microsoft, which was never the core of its business to being with, to focus even more on search and social networking, and its increasing competition with Facebook.

This shift in emphasis is reflected in some notable departures, as well as in a reorganization of the division that oversees the development of Google Apps, which include online office productivity tools that compete with Microsoft Office. Google continues to add functionality to Google Apps, but most of the functionality has either been in the works for years, or borrows from other existing products such as Google+.

Google Apps for businesses includes Web-based word processing, spreadsheet and presentation applications that the company hosts on its own computers and offers to companies for $50 a user, per year. The suite became popular among smaller businesses looking to transition from Microsoft Office software, which is hosted on company computers and requires maintenance from an IT staff.

Google Apps has had some churn to its core leadership as the company evolves under CEO Larry Page, including the loss of Dave Girouard as vice president of Apps and president of Google’s Enterprise business. Girouard, who joined Google in 2004, oversaw the development and launch of Apps for businesses. He left April 6 and no successor has been named.

Google
Amit Singh

A source familiar with Google Apps told CIO Journal: “I was personally shocked to see Dave G leave. That was his baby, and he was so invested in it.”

Girouard himself downplayed his exit in an e-mail to CIO Journal: “Google has an amazingly deep bench and the Enterprise biz has never been doing better.” Girouard left to launch a startup.

Other key Google Apps employees have also left the company or been reassigned to other projects. Matt Glotzbach, a product management director at Google Apps who was often the public face of the suite when Girouard wasn’t available, became managing director of Google’s YouTube unit in Europe last June. Apps also lost its top two Google public relations managers. Mike Nelson moved to Japan to lead Google’s public relations team there last year. Andrew Kovacs left earlier this year to run public relations for Sequoia Capital.

Tom Sarris, who replaced Kovacs three months ago as the public relations manager for Google Apps, told CIO Journal in early April he has not yet met with Sundar Pichai, who oversees the Google Apps business, among other responsibilities.

The executive exodus at Apps follows a restructuring of the Google Enterprise business under Page. Last summer, Page split the Google Enterprise business into two units — an Apps unit uniting Google’s consumer and business product teams, and another unit that focuses exclusively on selling Apps to businesses. Under this change, Girouard reported to Pichai, who manages the Google Chrome and Apps businesses. Amit Singh, responsible for sales of Apps to businesses, reports to Nikesh Arora, senior vice president and chief business officer at the company.

The split may seem confusing, but Singh told CIO Journal Page restructured the enterprise business to help the Apps product teams focus on development, leaving Singh and his team to sell the software to businesses.

To be sure, Apps appears to be in solid shape today. More than 4 million businesses rely on Google software to support their collaboration efforts, though Google said only hundreds of thousands of those companies are paying customers. The company in the last few months secured two large, paid contracts, including BBVA bank, which will put its 110,000 employees on Apps this year, and healthcare provider Roche Group, which agreed to put its 90,000 workers on the software.

And customers appear to be pleased with the software, which gained over 200 features in 2011 alone. Ahold, a large retailer based in Amsterdam, has been using Google Apps for its 55,000 employees in Europe and the U.S. since 2010, according to a company spokesman. Joe Fuller, CIO for Dominion Enterprises, said he has been pleased with his Apps implementation since he moved his 4,000 employees from Microsoft Office to Apps this year.

Google’s Singh said Girouard essentially incubated Apps as an enterprise business within Google. But now  the company is focusing on scaling the business. “Instead of seeing one large [customer] name a quarter, you’ll start to see several a quarter.” Singh also told CIO Journal Google would consider sensible acquisitions to prop up the Apps business.

Even so, the recent Apps management and stewardship changes are accompanied by a subtle shift in Google’s focus. Google’s application portfolio has broadened since Apps were introduced to include products such as Chrome and Android, which are key to the company’s mobile ambitions. When Google published Page’s update on its business for investors last week, Page touted products primed to fuel Google’s advertising revenues, including search, Android mobile software, Chrome, and Google+, the company’s new social network.

Page didn’t address the momentum of the Google Apps suite, and only referenced Gmail, the Web-based e-mail application that forms the core of Apps’ communications for businesses, as an afterthought: “And our enterprise customers love it too. Over 5,000 new businesses and educational establishments now sign up every day.”

For now, Google is still adding functionality to Google Apps. The company recently launched an archiving application that had been in development for years. A source familiar with Google Apps’ product road map said Apps users can expect the company to integrate Google+ social functionality with Google Apps over the course of 2012. Google could also more tightly integrate Apps with notebook computers based on its Chrome Operating System, the source said.

Google’s Enterprise business has historically only accounted for roughly 3% of the company’s annual revenues, with the lion’s share provided by advertising.

Microsoft has also developed a Web-based version of its Office suite, called Office 365. The suite has drawn favorable reviews from users and analysts, and is beginning to win some customers from Google. Chandris Hotels and Resorts and beauty care company Naturally Me recently said they chose Microsoft over Google Apps.

IDC analyst Melissa Webster, who talks to customers of both Apps and Office software, says more customers could join that exodus, especially if Google finds itself challenged in areas it considers more strategic, such as social, search and advertising.

“I could see Google Apps de-emphasized, or just not funded that aggressively,” Webster told CIO Journal. On the other hand, “Microsoft is and has always been an enterprise software vendor — they’re in it and committed for the long haul, that’s their DNA and Office is a strategic product line,” Webster noted.

Correction: CIO Journal incorrectly listed Contoso as a company that picked Microsoft over Google Apps. We regret the error.

0
Your rating: None

Is this level worth the 50p?
Remember when Google made a big deal of running Bastion in Chrome? They made the opposite of that fuss with Mini Ninjas, Io Interactives’s cute little NinjSim. I mean, I only noticed when I was poking around the Chrome store for a browser add-on. I clicked it expecting a webbified version of the game, but as far as I can tell it’s not been fiddled with in the slightest. In less than a minute, I was playing the first level of the game, full screen, with no obvious technical issues. Oh, apart from the bizarre payment model.
(more…)

0
Your rating: None

New submitter rescrv writes "Key-value stores (like Cassandra, Redis and DynamoDB) have been replacing traditional databases in many demanding web applications (e.g. Twitter, Google, Facebook, LinkedIn, and others). But for the most part, the differences between existing NoSQL systems come down to the choice of well-studied implementation techniques; in particular, they all provide a similar API that achieves high performance and scalability by limiting applications to simple operations like GET and PUT.

HyperDex, a new key-value store developed at Cornell, stands out in the NoSQL spectrum with its unique design. HyperDex employs a unique multi-dimensional hash function to enable efficient search operations — that is, objects may be retrieved without using the key (PDF) under which they are stored. Other systems employ indexing techniques to enable search, or enumerate all objects in the system. In contrast, HyperDex's design enables applications to retrieve search results directly from servers in the system. The results are impressive. Preliminary benchmark results on the project website show that HyperDex provides significant performance improvements over Cassandra and MongoDB. With its unique design, and impressive performance, it seems fittng to ask: Is HyperDex the start of NoSQL 2.0?"


Share on Google+

Read more of this story at Slashdot.

0
Your rating: None

snydeq writes "Deep End's Paul Venezia sees few business IT situations that could make good use of full cloud storage services, outside of startups. 'As IT continues in a zigzag path of figuring out what to do with this "cloud" stuff, it seems that some companies are getting ahead of themselves. In particular, the concept of outsourcing storage to a cloud provider puzzles me. I can see some benefits in other cloud services (though I still find the trust aspect difficult to reconcile), but full-on cloud storage offerings don't make sense outside of some rare circumstances.'"


Share on Google+

Read more of this story at Slashdot.

0
Your rating: None

judgecorp writes "The OpenStack open source cloud project has removed Hyper-V from its infrastructure as a service (IaaS) framework, saying Microsoft's support for its hypervisor technology is 'broken.' This will embarass Microsoft, as major partners such as Dell and HP support OpenStack, along with service providers such as Internap." Adds reader alphadogg, this "means the code will be removed when the next version of OpenStack, called Essex, is released in the second quarter."


Share on Google+

Read more of this story at Slashdot.

0
Your rating: None