Skip navigation
Help

large systems

warning: Creating default object from empty value in /var/www/vhosts/sayforward.com/subdomains/recorder/httpdocs/modules/taxonomy/taxonomy.pages.inc on line 33.

When Joel Spolsky, my business partner on Stack Overflow and Stack Exchange, asked me what I wanted to do after I left Stack Exchange, I distinctly remember mentioning Aaron Swartz. That's what Aaron was to us hackers: an exemplar of the noble, selfless behavior and positive action that all hackers aspire to – but very few actually achieve.

And now, tragically, Aaron is gone at the tender age of 26. He won't be achieving anything any more.

I never knew Aaron, but I knew Aaron.

Aaron-swartz-stack-overflow

Most of all, I am disappointed.

I'm deeply disappointed in myself, for not understanding just how bitterly unfair the government charges were against Aaron. Perhaps the full, grotesque details couldn't be revealed for a pending legal case. But we should have been outraged. I am gutted that I did not contribute to his defense in any way, either financially or by writing about it here. I blindly assumed he would prevail, as powerful activists on the side of fairness, openness, and freedom are fortunate enough to often do in our country. I was wrong.

I'm disappointed in our government, for going to such lengths to make an example of someone who was so obviously a positive force. Someone who actively worked to change the world for the better in everything he did, starting from the age of 12. There was no evil in this man. And yet the absurd government case against him was cited by his family as directly contributing to his death.

I'm frustrated by the idea that martyrdom works. The death of Aaron Swartz is now turning into an effective tool for change, a rallying cry, proving the perverse lesson that nobody takes an issue seriously until a great person dies for the cause. The idea that Aaron killing himself was a viable strategy, more than going on to prevail in this matter and so many more in his lifetime, makes me incredibly angry.

But also, I must admit that I am a little disappointed in Aaron. I understand that depression is a serious disease that can fell any person, however strong. But he chose the path of the activist long ago. And the path of the activist is to fight, for as long and as hard as it takes, to effect change. Aaron had powerful friends, a powerful support network, and a keen sense of moral cause that put him in the right. That's how he got that support network of powerful friends and fellow activists in the first place.

It is appropriate to write about Aaron on Martin Luther King day, because he too was a tireless activist for moral causes.

I hope you are able to see the distinction I am trying to point out. In no sense do I advocate evading or defying the law, as would the rabid segregationist. That would lead to anarchy. One who breaks an unjust law must do so openly, lovingly, and with a willingness to accept the penalty. I submit that an individual who breaks a law that conscience tells him is unjust, and who willingly accepts the penalty of imprisonment in order to arouse the conscience of the community over its injustice, is in reality expressing the highest respect for law.

Let's be clear that the penalty in Aaron's case was grossly unfair, bordering on corrupt. I've been a part of exactly one trial, but I can't even imagine having the full resources of the US Government brought to bear against me, with extreme prejudice, for a year or more. His defense was estimated to cost millions. The idea that such an engaged citizen would be forever branded a felon – serving at least some jail time and stripped of the most fundamental citizenship right, the ability to vote – must have weighed heavily on Aaron. And Aaron was no stranger to depresson, having written about it on his blog many times, even penning a public will of sorts on his blog all the way back in 2002.

I think about ragequitting a lot.

Rage Quit, also seen as RageQuit in one word, is Internet slang commonly used to describe the act of suddenly quitting a game or chatroom after either an argument, extreme frustration, or loss of the game.

At least one user ragequits Stack Exchange every six months, because our rules are strict. Some people don't like rules, and can respond poorly when confronted by the rules of the game they choose to play. It came up often enough that we had to create even more rules to deal with it. I was forced to think about ragequitting.

I was very angry with Mark Pilgrim and _why for ragequitting the Internet, because they also took all their content offline – they got so frustrated that they took their ball and went home, so nobody else could play. How incredibly rude. Ragequitting is childish, a sign of immaturity. But it is another thing entirely to play the final move and take your own life. To declare the end of this game and all future games, the end of ragequitting itself.

I say this not as a person who wishes to judge Aaron Swartz. I say it as a fellow gamer who has also considered playing the same move quite recently. To the point that I – like Aaron himself, I am sure – was actively researching it. But the more I researched, the more I thought about it, the more it felt like what it really was: giving up. And the toll on friends and family would be unimaginably, unbearably heavy.

What happened to Aaron was not fair. Not even a little. But this is the path of the activist. The greater the injustice, the greater wrong undone when you ultimately prevail. And I am convinced, absolutely and utterly convinced, that Aaron would have prevailed. He would have gone on to do so many other great things. It is our great failing that we did not provide Aaron the support network he needed to see this. All we can do now is continue the mission he started and lobby for change to our corrupt government practices of forcing plea bargains.

It gets dark sometimes. I know it does. I'm right there with you. But do not, under any circumstances, give anyone the satisfaction of seeing you ragequit. They don't deserve it. Play other, better moves – and consider your long game.

[advertisement] Stack Overflow Careers matches the best developers (you!) with the best employers. You can search our job listings or create a profile and even let employers find you.

0
Your rating: None

An anonymous reader writes "Graphene once again proves that it is quite possibly the most miraculous material known to man, this time by making saltwater drinkable. The process was developed by a group of MIT researchers who realized that graphene allowed for the creation of an incredibly precise sieve. Basically, the regular atomic structure of graphene means that you can create holes of any size, for example the size of a single molecule of water. Using this process scientist can desalinate saltwater 1,000 times faster than the Reverse Osmosis technique."


Share on Google+

Read more of this story at Slashdot.

0
Your rating: None

In computing, a hypervisor, also called virtual machine manager (VMM), is one of many hardware virtualization techniques allowing multiple operating systems, termed guests, to run concurrently on a host computer. It is so named because it is conceptually one level higher than a supervisory program. The hypervisor presents to the guest operating systems a virtual operating platform and manages the execution of the guest operating systems. Multiple instances of a variety of operating systems may share the virtualized hardware resources. Hypervisors are installed on server hardware whose only task is to run guest operating systems.

The term can be used to describe the interface provided by the specific cloud computing functionality infrastructure as a service (IaaS).[1][2]

The term "hypervisor" was first used in 1965, referring to software that accompanied an IBM RPQ for the IBM 360/65. It allowed the model IBM 360/65 to share its memory: half acting like a IBM 360; half as an emulated IBM 7080. The software, labeled "hypervisor," did the switching between the 2 modes on split time basis. The term hypervisor was coined as an evolution of the term "supervisor," the software that provided control on earlier hardware.[3][4]

Classification

Robert P. Goldberg classifies two types of hypervisor:[5]

  • Type 1 (or native, bare metal) hypervisors run directly on the host's hardware to control the hardware and to manage guest operating systems. A guest operating system thus runs on another level above the hypervisor.
This model represents the classic implementation of virtual machine architectures; the original hypervisor was CP/CMS, developed at IBM in the 1960s, ancestor of IBM's z/VM. A modern equivalent of this is the Citrix XenServer, VMware ESX/ESXi, and Microsoft Hyper-V hypervisor.
  • Type 2 (or hosted) hypervisors run within a conventional operating system environment. With the hypervisor layer as a distinct second software level, guest operating systems run at the third level above the hardware. KVM and VirtualBox are examples of Type 2 hypervisors.

In other words, Type 1 hypervisor runs directly on the hardware; a Type 2 hypervisor runs on another operating system, such as FreeBSD[6] or Linux[7].

Note: Microsoft Hyper-V (released in June 2008)[8] exemplifies a type 1 product that can be mistaken for a type 2. Both the free stand-alone version and the version that is part of the commercial Windows Server 2008 product use a virtualized Windows Server 2008 parent partition to manage the Type 1 Hyper-V hypervisor. In both cases the Hyper-V hypervisor loads prior to the management operating system, and any virtual environments created run directly on the hypervisor, not via the management operating system.

Hyperviseur.png

Mainframe origins

The first hypervisor providing full virtualization, IBM's one-off research CP-40 system, began production use in January 1967, and became the first version of IBM's CP/CMS operating system. CP-40 ran on a S/360-40 that was modified at the IBM Cambridge Scientific Center to support Dynamic Address Translation, a key feature that allowed virtualization. Prior to this time, computer hardware had only been virtualized enough to allow multiple user applications to run concurrently (see CTSS and IBM M44/44X). With CP-40, the hardware's supervisor state was virtualized as well, allowing multiple operating systems to run concurrently in separate virtual machine contexts.

Programmers soon re-implemented CP-40 (as CP-67) for the IBM System/360-67, the first production computer-system capable of full virtualization. IBM first shipped this machine in 1966; it included page-translation-table hardware for virtual memory, and other techniques that allowed a full virtualization of all kernel tasks, including I/O and interrupt handling. (Note that its "official" operating system, the ill-fated TSS/360, did not employ full virtualization.) Both CP-40 and CP-67 began production use in 1967. CP/CMS was available to IBM customers from 1968 to 1972, in source code form without support.

CP/CMS formed part of IBM's attempt to build robust time-sharing systems for its mainframe computers. By running multiple operating systems concurrently, the hypervisor increased system robustness and stability: Even if one operating system crashed, the others would continue working without interruption. Indeed, this even allowed beta or experimental versions of operating systems – or even of new hardware[9] – to be deployed and debugged, without jeopardizing the stable main production system, and without requiring costly additional development systems.

IBM announced its System/370 series in 1970 without any virtualization features, but added them in the August 1972 Advanced Function announcement. Virtualization has been featured in all successor systems. (All modern-day (as of 2009[update]) IBM mainframes, such as the zSeries line, retain backwards-compatibility with the 1960s-era IBM S/360 line.) The 1972 announcement also included VM/370, a reimplementation of CP/CMS for the S/370. Unlike CP/CMS, IBM provided support for this version (though it was still distributed in source code form for several releases). VM stands for Virtual Machine, emphasizing that all, and not just some, of the hardware interfaces are virtualized. Both VM and CP/CMS enjoyed early acceptance and rapid development by universities, corporate users, and time-sharing vendors, as well as within IBM. Users played an active role in ongoing development, anticipating trends seen in modern open source projects. However, in a series of disputed and bitter battles, time-sharing lost out to batch processing through IBM political infighting, and VM remained IBM's "other" mainframe operating system for decades, losing to MVS. It enjoyed a resurgence of popularity and support from 2000 as the z/VM product, for example as the platform for Linux for zSeries.

As mentioned above, the VM control program includes a hypervisor-call handler which intercepts DIAG ("Diagnose") instructions used within a virtual machine. This provides fast-path non-virtualized execution of file-system access and other operations. (DIAG is a model-dependent privileged instruction, not used in normal programming, and thus is not virtualized. It is therefore available for use as a signal to the "host" operating system.) When first implemented in CP/CMS release 3.1, this use of DIAG provided an operating system interface that was analogous to the System/360 SVC ("supervisor call") instruction, but that did not require altering or extending the system's virtualization of SVC.

In 1985 IBM introduced the PR/SM hypervisor to manage logical partitions (LPAR).

UNIX and Linux servers

Several factors led to a resurgence around 2005[10] in the use of virtualization technology among UNIX and Linux server vendors:

  • expanding hardware capabilities, allowing each single machine to do more simultaneous work
  • efforts to control costs and to simplify management through consolidation of servers
  • the need to control large multiprocessor and cluster installations, for example in server farms and render farms
  • the improved security, reliability, and device independence possible from hypervisor architectures
  • the ability to run complex, OS-dependent applications in different hardware or OS environments

Major UNIX vendors, including Sun Microsystems, HP, IBM, and SGI, have been selling virtualized hardware since before 2000. These have generally been large systems with hefty, server-class price-tags (in the multi-million dollar range at the high end), although virtualization is also available on some mid-range systems, such as IBM's System-P servers, Sun's CoolThreads T1000, T2000 and T5x00 servers and HP Superdome series.

Multiple host operating systems have been modified[by whom?] to run as guest OSes on Sun's Logical Domains Hypervisor. As of late 2006[update], Solaris, Linux (Ubuntu and Gentoo), and FreeBSD have been ported to run on top of Hypervisor (and can all run simultaneously on the same processor, as fully virtualized independent guest OSes). Wind River "Carrier Grade Linux" also runs on Sun's Hypervisor.[11] Full virtualization on SPARC processors proved straightforward: since its inception in the mid-1980s Sun deliberately kept the SPARC architecture clean of artifacts that would have impeded virtualization. (Compare with virtualization on x86 processors below.)[12]

HP calls its technology to host multiple OS technology on its Itanium powered systems (Integrity) "Integrity Virtual Machines" (Integrity VM). Itanium can run HP-UX, Linux, Windows and OpenVMS. Except for OpenVMS, to be supported in a later release, these environments are also supported as virtual servers on HP's Integrity VM platform. The HP-UX operating system hosts the Integrity VM hypervisor layer which allows for many important features of HP-UX to be taken advantage of and provides major differentiation between this platform and other commodity platforms - such as processor hotswap, memory hotswap, and dynamic kernel updates without system reboot. While it heavily leverages HP-UX, the Integrity VM hypervisor is really a hybrid that runs on bare-metal while guests are executing. Running normal HP-UX applications on an Integrity VM host is heavily discouraged[by whom?], because Integrity VM implements its own memory management, scheduling and I/O policies that are tuned for virtual machines and are not as effective for normal applications. HP also provides more rigid partitioning of their Integrity and HP9000 systems by way of VPAR and nPar technology, the former offering shared resource partitioning and the later offering complete I/O and processing isolation. The flexibility of virtual server environment (VSE) has given way to its use more frequently in newer deployments.[citation needed]

IBM provides virtualization partition technology known as logical partitioning (LPAR) on System/390, zSeries, pSeries and iSeries systems. For IBM's Power Systems, the Power Hypervisor (PowerVM) functions as a native (bare-metal) hypervisor and provides EAL4+ strong isolation between LPARs. Processor capacity is provided to LPARs in either a dedicated fashion or on an entitlement basis where unused capacity is harvested and can be re-allocated to busy workloads. Groups of LPARs can have their processor capacity managed as if they were in a "pool" - IBM refers to this capability as Multiple Shared-Processor Pools (MSPPs) and implements it in servers with the POWER6 processor. LPAR and MSPP capacity allocations can be dynamically changed. Memory is allocated to each LPAR (at LPAR initiation or dynamically) and is address-controlled by the POWER Hypervisor. For real-mode addressing by operating systems (AIX, Linux, IBM i), the POWER processors (POWER4 onwards) have architected virtualization capabilities where a hardware address-offset is evaluated with the OS address-offset to arrive at the physical memory address. Input/Output (I/O) adapters can be exclusively "owned" by LPARs or shared by LPARs through an appliance partition known as the Virtual I/O Server (VIOS). The Power Hypervisor provides for high levels of reliability, availability and serviceability (RAS) by facilitating hot add/replace of many parts (model dependent: processors, memory, I/O adapters, blowers, power units, disks, system controllers, etc.)

Similar trends have occurred with x86/x86_64 server platforms, where open-source projects such as Xen have led virtualization efforts. These include hypervisors built on Linux and Solaris kernels as well as custom kernels. Since these technologies span from large systems down to desktops, they are described in the next section.

PCs and desktop systems

Interest in the high-profit server-hardware market sector has led to the development of hypervisors for machines using the Intel x86 instruction set, including for traditional desktop PCs. One of the early PC hypervisors, the commercial-software VMware, debuted in 1998.

The x86 architecture used in most PC systems poses particular difficulties to virtualization. Full virtualization (presenting the illusion of a complete set of standard hardware) on x86 has significant costs in hypervisor complexity and run-time performance. Starting in 2005, CPU vendors have added hardware virtualization assistance to their products, for example: Intel's Intel VT-x (codenamed Vanderpool) and AMD's AMD-V (codenamed Pacifica). These extensions address the parts of x86 that are difficult or inefficient to virtualize, providing additional support to the hypervisor. This enables simpler virtualization code and a higher performance for full virtualization.

An alternative approach requires modifying the guest operating-system to make system calls to the hypervisor, rather than executing machine I/O instructions which the hypervisor then simulates. This is called paravirtualization in Xen, a "hypercall" in Parallels Workstation, and a "DIAGNOSE code" in IBM's VM. VMware supplements the slowest rough corners of virtualization with device drivers for the guest. All are really the same thing, a system call to the hypervisor below. Some microkernels such as Mach and L4 are flexible enough such that "paravirtualization" of guest operating systems is possible.

In June 2008 Microsoft delivered a new Type-1 hypervisor called Hyper-V (codenamed "Viridian" and previously referred to as "Windows Server virtualization"); the design features OS integration at the lowest level.[13] Versions of the Windows operating system beginning with Windows Vista include extensions to boost performance when running on top of the Hyper-V hypervisor.

Embedded systems

As of 2009[update] virtual machines have started to appear in embedded systems, such as mobile phones. This provides a high-level operating-system interface for application programming, such as Linux or Microsoft Windows, while at the same time maintaining traditional real-time operating system (RTOS) APIs. The low-level RTOS environments need to be retained for legacy support, and because the real-time capabilities of high-level OSes are insufficient for many embedded applications.

Embedded hypervisors must therefore have real-time capability, a design criterion not present for hypervisors used in other domains. The resource-constrained nature of many embedded systems, especially battery-powered mobile systems, imposes a further requirement for small memory-size and low overhead. Finally, in contrast to the ubiquity of the x86 architecture in the PC world, the embedded world uses a wider variety of architectures. Support for virtualization requires memory protection (in the form of a memory management unit or at least a memory protection unit) and a distinction between user mode and privileged mode, which rules out most microcontrollers. This still leaves x86, MIPS, ARM and PowerPC as widely deployed architectures on medium- to high-end embedded systems.

As manufacturers of embedded systems usually have the source code to their operating systems, they have less need for full virtualization in this space. Instead, the performance advantages of paravirtualization make this usually the virtualization technology of choice. Nevertheless, ARM has recently added full virtualization support as an IP option and has included it in their latest high end processor codenamed Eagle.

Other differences between virtualization in server/desktop and embedded environments include requirements for efficient sharing of resources across virtual machines, high-bandwidth, low-latency inter-VM communication, a global view of scheduling and power management, and fine-grained control of information flows.[14]

Security implications

The use of hypervisor technology by malware and rootkits installing themselves as a hypervisor below the operating system can make them more difficult to detect because the malware could intercept any operations of the operating system (such as someone entering a password) without the antivirus software necessarily detecting it (since the malware runs below the entire operating system). Implementation of the concept has allegedly occurred in the SubVirt laboratory rootkit (developed jointly by Microsoft and University of Michigan researchers[15]) as well as in the Blue Pill malware package. However, such assertions have been disputed by others who claim that it would indeed be possible to detect the presence of a hypervisor-based rootkit.[16]

In 2009, researchers from Microsoft and North Carolina State University demonstrated a hypervisor-layer anti-rootkit called Hooksafe that can provide generic protection against kernel-mode rootkits.[17]

See also

0
Your rating: None

rain cloudWe recently moved from Amazon on-demand “cloud” hosting to our own dedicated servers.  It took about three months to order and set up the new servers versus a few minutes to get servers on Amazon.  However, the new servers are 2.5X faster and so far, more reliable.

We love Amazon for fostering development and innovation.  Cloud computing systems are great at getting you new servers.  This helps a lot when you are trying to innovate because you can quickly get new servers for your new services. If you are in a phase of trying new things, cloud hosts will help you.

Cloud hosts also help a lot when you are testing.  It’s amazing how many servers it takes to run an Internet service.  You don’t just need production systems.  You need failover systems.  You need development systems.  You need staging/QA systems.  You will need a lot of servers, and you may need to go to a cloud host.

However, there are problems with cloud hosting that emerge if you need high data throughput.  The problems aren’t with the servers but instead, with storage and networking.  To see why, let’s look at how a cloud architecture differs from a local box architecture.  You can’t directly attach each storage location to the box that it servers.  You have to use network attached storage.

DEDICATED ARCHITECTURE:  Server Box -> bus or lan or SAN -> Storage

CLOUD ARCHITECTURE:  Server Box -> Mesh network -> Storage cluster with network replication

1) Underlying problem:  Big data, slow networks

Network attached storage becomes a problem because there is a fundamental mismatch between networking and storage.  Storage capacity almost doubles every year.  Networking speed grows by a factor of ten about every 10 years – 100 times lower.  The net result is that storage gets much bigger than network capacity, and it takes a really long time to copy data over a network.  I first heard this trend analyzed by John Landry, who called it “Landry’s law.”  In my experience, this problem has gotten to the point where even sneakernet (putting on sneakers and carrying data on big storage media) cannot save us because after you lace up your sneakers, you have to copy the data OVER A NETWORK to get it onto the storage media and then copy it again to get it off.  When we replicated the Assembla data to the new datacenter, we realized that it would be slower to do those two copies than to replicate over  the Internet, which is slower than sneakernet for long distance transport but only requires one local network copy.

2) Mesh network inconsistency

The Internet was designed as a hub and spoke network, and that part of it works great.  When you send a packet up from your spoke, it travels a predictable route through various hubs to its destination.  When you plug dedicated servers into the Internet, you plug a spoke into the hub, and it works in the traditional way.  The IP network inside a cloud datacenter is more of a “mesh.”  Packets can take a variety of routes between the servers and the storage.  The mesh component is vulnerable to both packet loss and capacity problems.  I can’t present any technical reason why this is true, but in our observation, it is true.  We have seen two different issues:

* Slowdowns and brownouts:  This is a problem at both Amazon and GoGrid, but it is easier to see at Amazon.  Their network, and consequently their storage, has variable performance, with slow periods that I call “brownouts.”

* Packet loss:  This is related to the capacity problems as routers will throw away packets when they are overloaded.  However, the source of the packet loss seems to be much harder to debug in a mesh network.  We see these problems on the GoGrid network, and their attempts to diagnose it are often ineffectual.

3) Replication stoppages

The second goal of cloud computing is to provide high availability. The first goal is to never lose data.  When there is a failure in the storage cluster, the first goal (don’t lose data) kicks in and stomps on the second goal (high availability).  Systems will stop accepting new data and make sure that old data gets replicated.  Network attached storage will typically start replicating data to a new node.  It may either refuse new data until it can be replicated reliably, or it will absorb all network capacity and block normal operation in the mesh.

Note that in a large complex systems, variations in both network speed and storage capacity will follow a power law distribution.  This happens "chaotically."  When the variation reaches a certain low level of performance, the system fails because of the replication problem. 

I think that we should be able to predict the rate of major failures by observing the smaller variations and extrapolating them with a power law.  Amazon had  a major outage in April 2011. Throughout the previous 18 months, they had performance brownouts, and I think the frequency of one could be predicted from the other.

CONCLUSION

So, if your application is storage intensive and high availability, you must either:

1) Design it so that lots of replication is running all of the time, and you can afford to lose access to any specific storage node.  This places limits on the speed that your application can absorb data because you need to reserve a big percentage of scarce network capacity for replication.  So, you will have only a small percentage of network capacity available to for absorbing external data.  However, it is the required architecture for very large systems.  It  works well if you have a high ratio of output to input, since output just uses the replicated data rather than adding to it.

If you try this replication strategy, you will need to deal with two engineering issues.  First, you will think through replication specifically for your application.  There are many new database architectures that make this tradeoff in various ways.  Each has strengths and weaknesses, so if you design a distributed system, you will probably end up using several of these new architectures.  Second, you will need to distribute across multiple mesh network locations. It's not enough just to have several places to get your data, in the same network neighborhood.  If there is a problem, the entire mesh will jam up.  Ask about this.

2) Use local storage

0
Your rating: None

My foreign policy is always war.
Three wise men once said, “Intergalactic, planetary. Planetary, intergalactic.” I think we all take solace from those words at some points in our life. For are we not all made of star stuff? Solar 2 is built on this species of Saganian wisdom. Coupled with some delightful planetary physics and a dash of good humour, it’s probably the best time you can have alone with a heavenly body. Um. That’s not what I mean. Um. Um.

(more…)

0
Your rating: None