Today, we are pleased to announce that the beta 5 of Archipel, named “Moon” is now available. Well, actually it was already available since sunday evening, but the announcement is done today because we were waiting to publish the first version of ANSOS in the same time. Some of you may ask themselves: “What is ANSOS?”.
ANSOS
ANSOS stands for Archipel Node Stateless OS. It’s a Live Linux OS based on oVirt Node. oVirt is what you use when you want to deploy a new hypervisor in an oVirt environment. ANSOS is what you use when you want to deploy a new Archipel hypervisor. The great thing with ANSOS is that it is not a separate project or a fork. All the changes I made in oVirt to make Archipel working have been merged into the oVirt Node master. This ensure a great support from Archipel team, oVirt Team and community. The major difference with previous version of oVirt Node and ANSOS is that it allows to boot up in a completely stateless mode. You basically set a small bunch of kernel arguments when you start the OS to tell how and where to connect to a shared storage point for persistent data, and that is all. The embedded Archipel agent will do the rest, retrieving default configuration and specific configuration from this shared mount, launch eventual post script to let you fine tune the OS etc.
It is available now, and you can download it from here.
Beta 5 “Moon”
Archipel itself has been severely upgraded. There are a lot of bug fixed and new features. You can find the full change log here. The main ones are:
- Auto Grouping: Now hypervisors and virtual machines are automatically added in Shared Roster groups (need XMLRPC API for now)
- VM Parking: Store XML description in a virtual parking. The VM will be set offline and no agent will run it. Later you can remove it from parking and restore it on any hypervisor you like
- Full support for Character Devices: console, serial, parallels
- Multiple entities selection in the roster
- Extend libvirt API support in model (<video>, <os>, <clock>, <hostdev>)
- Support for creating drives in transient mode, shareable mode or read only mode
- Health module now computes stats in a web worker (thread)
- Health module now displays the amount of Shared Memory
- Tables with potentially tons of data (like users) are now using a new lazy loading API
- Improve keyboard support in VNC (french) and add some new layouts (“es”, “no”, “hu”)
- Preliminary support for configuring SPICE screen (no UI)
- Support of macvtap per Network and/or per VM’s NIC
- Improved Live migration error handling and sanity checks
- Great performances improvements by reducing the amount of XMPP stanza sent
- New versions of all frameworks (Cappuccino, StropheCappuccino, VNCCappuccino, TNKit, LPKit, GrowlCappuccino)
- New CLI tool named archipel-command that allows to send raw stanzas to Archipel entities
- New library named archipelcore.scriptutils to make your life easier when you create script to manage your Archipel platform
And obviously tons of bug fixes and other minor improvements. Everthing is available here.
The update is done as usual if you used the beta 4 by:
# easy_install -U archipel-agent
Note that you need to delete /var/lib/archipel/statscollection.sqlite3 before restarting Archipel or you’ll get errors in Health module.
We hope you will enjoy this release.
I notice that most of people don’t know how to deal with errors in Archipel. This post will try to help people to know how to manage problems in Archipel, track the origin of the errors, and how to fix them, or at least being able to report a correct issue.
First let’s see how I organize my desktop. This is my personal preference, but I guess it’s a good starting point. I have two screens, and this is sincerely the bare minimum for me.
Screen 1:
Screen 2:
As you can see, on the first screen, I have my browser, with debugger opened, the source code (this is obviously optional if you are just a user) and on the second screen I have opened two terminals. One connected to the hypervisors through SSH, with the archipel.log displayed with tail (I use one tab per hypervisor) and another one, in my local computer, to send update.
As you can see, to update the archipel agent code, a simple scp + restart archipel is sufficient because I have the source installed with buildAgent -d. This allows to optimize the code’n'try way. I also usually have another terminal SSH’ed to the hypervisors to send simple commands, or fix some stuff quickly. If you use Mac OS X, you can create a Terminal profile to open this workspace with one click.
There are basically two main errors users are complaining about.
- Their hypervisors are offline
- They get some 501 errors
Usually, there are only few things that could cause these issues. it’s mostly always due to :
- some missing library like python-libvirt or xmpppy or stuff like that
- a misconfiguration in archipel.conf
- A problem with the ejabberd configuration
The basics
When you are debugging Archipel, you need to know that most of the errors are logged in the archipel.log. In case they are not logged, there is great chance you run into a new issue we are not aware of. To be sure you will see every kind of errors, do not start archipel with the init script! it will redirect all output to /dev/null. You should start and restart archipel like this
# killall runarchipel; runarchipel; tailf /var/log/archipel/archipel.log
Do not hesitate to grep! for example:
# killall runarchipel; runarchipel; tailf /var/log/archipel/archipel.log | grep -i error # only displays error lines
# killall runarchipel; runarchipel; tailf /var/log/archipel/archipel.log | grep -i “uuid@fqdn” # only displays logs from a the vm with given UUID
Be creative, and try to always filter the log, because it can be very verbose. most of the functions prefixes the logs with an identifier. for example, if you want to get logs for only migration related stuff, grep on “MIGRATION” or only for VMParking feature, grep on “VMPARKING”.
This will help you to track down where your problem comes from.
The 501 errors
You have to note that 501 errors are not logged into archipel.log! Errors 501 mean the agent hasn’t react to the command, so obviously, nothing will be logged. But, it is logged on the client:
and in the browser console:
The Browser console is a capital tool. It contains the very XMPP stanzas that have triggered the errors. For instance, if we looked at the second error message, we see that the stanza is:
<iq xmlns='jabber:client' from='hypervisor@ramucho/ramucho' to='admin@ramucho/ArchipelController' id='4771'type='error'>
<query xmlns='archipel:hypervisor:health'>
<archipel xmlns='archipel:hypervisor:health' action='logs' limit='50'/>
</query>
<error code='501' type='cancel'>
<feature-not-implemented xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/>
<text xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'>
The feature requested is not implemented by the recipient or server and therefore cannot be processed.
</text>
</error>
</iq>
Which means that the hypervisor hasn’t computed the command archipel:hypervisor:health->logs. In that case, it’s easy to understand because I have disabled the health module. So the XMPP stanza has not be handled.
When this happens involuntarily, this means that during startup, an error has occurred. Let’s remove manually a required token like “vnc_certificate_file” which is required by the VNC module. When I start the agent, greping on “ERROR” I will have in the archipel.log:
These message is pretty explicit. I have two virtual machines —653b…@ramucho and 2590…@ramucho — who are complaining about the impossibility to load the VNC plugin because of the missing token “vnc_certificate_file”. This will not prevent the agent to start, but if you try from the UI to access the VNC screen, you will get 501 errors. That’s pretty simple.
The hypervisor offline error
The first thing to do is to check if the hypervisor is really offline. To do so, ask the ejabberd server:
# ejabberdctl connected_users your.fqdn.com
It will list the accounts that are actually connected. This often help to track down the basis of the error. If you see your hypervisor online, well you certainly made a typo when adding it to your roster while entering the JID. If it’s offline, you’ll see in the log why! There are plenty of possible cause, most of them are logged. If it’s a more critical issues, you will certainly a exception trace printed on STDERR (remember we have manually started archipel).
Conclusion
It’s important to track down your eventual problems before reporting an issue, or asking for help on the IRC channel. Most of the time, these errors are extremely easy to fix, but the user report them like “it doesn’t work”. It doesn’t help that much, and we loose a large amount of time to explain this. So before reporting any problem:
- Check the archipel.log
- Check the JS console
- Check your network connectivity
- Be sure you have not made any typo! (seriously, this happens quite often)
- Try to do simple test to exclude some test case
As a final note, we greatly appreciate users that send crash reports. But please, consider adding a very short description of what was your last action before the crash. Otherwise, depending on the browser, that trace may not be usable.
It’ been a while since the last post. A lot of stuff have been added to Archipel since. So I guess it’s time to explain what changed.
First of all, we worked hard on UI overall performance. Archipel has grown in complexity, and some modules were just to resource greedy. We have added some controls to be sure a given request is not already waiting for a response, and if it is, Archipel just replaces it with the new one, instead of receiving two answers, one useful, one ignored. The majority of these improvements have been implemented at the lowest possible level. We also worked on optimization of the permissions caching. The result is a really smoother user experience.
Also, all the possible very big tables, like the XMPP users one, are now using a lazy loading system. Archipel UI will only ask the small chucks of data it needs to display, and ask the rest later if needed. This allows you to have virtually any number of XMPP users you like, without impacting the performances. (there is still some places to optimize, though)
Now about features, we have several new stuff:
- Dynamic platform administrators: it’s now super easy to grant any user the full platform admin permissions: one click, real time.
- Support for bandwidth limitation per virtual machine.
- VMParking: You can “park” a virtual machine in a PubSub node. The VM doesn’t exist anymore anywhere. Then you can un-park it on any hypervisor you like. (you can also change the XML description of a parked VM)
- Support for character devices: manage the serial/parallels/channel devices for your VMs.
- Auto group: If you use XMLRPC API, you can set Archipel to automatically add VMs and hypervisors into shared roster groups.
- The dependency to python-magic, which caused a lot of installation problems, has been removed. A correct version is now embedded in the agent.
- Lot of bug fixes in the roster management
All these features are now available in the master head. But you have to note that you need to update everything. Older versions of agents/GUI are not compatible anymore with these changes.
Enjoy,
On 2011/11/24 at 17h45, Emmanuel Blindauer will give a talk about Archipel at french event JRES2011 in Toulouse. You certainly virtually meet him in #archipel IRC channel as Mooby. He will make an overall presentation about Archipel, what it does, how it does it, what are the advantages and the inconvenient (but there is no inconvenient). It will be a short talk, but if you are around Toulouse, feel free to go.
The beta 4 of Archipel, named “Vesta” is now available. It includes a lot of bug fixes, and we would like to thanks all the people who have sent crash reports and open issues. This has really helped us to track and fix the bugs. But it’s not only a maintenance release. It includes some cool new features. Here is the important ones:
- Easy support of additional admin accounts in GUI
- Manage existing libvirt domains directly from GUI
- Support for macvtap networking
- Support for multiple selection from the roster
- XML Editor for networks
- Support for inbound/outbound bandwidth limitation
- Display libvirt version and driver version of hypervisors
- Updated versions of external components
- New optimized push center
The full change log is available here.
As always, you can download the beta 4 of the GUI from the download page. The online version of the beta 4 is here.
To update the agent, run, as usual:
# easy_install -U archipel-agent
If you have already a previous beta installed, check carefully the log for missing conf tokens. If you need any help, ask us on the chat.
Hope you will enjoy it!
Until now, the most easy way to give virtual machines LAN connectivity is to use bridges. The standard workflow is to create a bridge first (manually or from Hypervisor Networks module), then log on your hypervisor, add a physical NIC in the bridge. This needs to shut down the network, configure some stuff etc. It’s pretty annoying and as libvirt doesn’t manage adding/removing NICs to/from bridges, most of the time, the Archipel’s Hypervisor Network module is pretty useless.
Well this time is over.
The Linux Kernel supports a new driver named “macvtap” since 2.6.34 and Libvirt supports it since 0.9.4 . Here is the definition:
“Macvtap is a new device driver meant to simplify virtualized bridged networking. It replaces the combination of the tun/tap and bridge drivers with a single module based on the macvlan device driver. A macvtap endpoint is a character device that largely follows the tun/tap ioctl interface and can be used directly by kvm/qemu and other hypervisors that support the tun/tap interface. The endpoint extends an existing network interface, the lower device, and has its own mac address on the same ethernet segment. Typically, this is used to make both the guest and the host show up directly on the switch that the host is connected to.” (source)
We are happy to announce that Archipel now supports this driver. It’s very easy to use. Select a hypervisor, go to the “Networks” tab, then create a network. Name it as you want, select the mode named “Bridge” or “Private”, and select the forward device (eth0 for instance):
The “Bridge” mode will act like a standard bridge. The virtual machines in the network will have a standard LAN access. The “Private” mode is the same, but it won’t be able to communicate to the rest of the VM on the host.
What you need to do on your hypervisor to support this feature is to load the “macvtap” driver
# modprobe macvtap
You also need to be sure the ethX you will use as forward device is not already in a bridge and you must use Libvirt 0.9.4+.
You may also notice there are more fields in the Network Edition Popover. You can now set the Inband / Outband traffic shaping! And last but not, least, we have added a manual XML editor for the networks, just like the Virtual Machine Definition:
All of this should be already available in the master at the time you read this. We hope you will enjoy it!
Last saturday, we were to the Open World Forum 2011, in Paris, to speak about Archipel. It was pretty interesting, but they were less people than the organizers expected. I don’t know for you guys, but here in France, in a Saturday morning at 11h, people are mostly sleeping :) Anyway, it was cool, the demo runs fine (only one crash!) and we were able to present everything we wanted to show. We also have been interviewed by Intelli-n.tv (the video is not published yet, we’ll update this entry when they will release it).
If you are interested, you can have a look at the keynote used for the presentation here.
The issue #130 of Linux Magazine has a great cover story about Archipel. There are 5 pages describing the solution in details and how to get it working (even if it is already a bit outdated as we have simplified a lot since the writing). If you haven’t this magazine, you can buy it here (plus, there are a lot of other interesting articles about virtualization in general).
The article is pretty nice and fair about Archipel, and we would like to thanks Andrej Radonic for it.
The beta 3.2 of Archipel has just been published. It is a minor release that fixes several bugs found in beta 3.1:
- In any case, the GUI should not crash anymore if problem occurs while reading a corrupted plist from XMPP Server
- Fix a bug with XMPP Server that doesn’t display users if using XMLRPC API mode
- Several other minor fixes.
# easy_install -U archipel-agent
Two versions in less than one week! Ok this one mainly contains only fixes, but also allow you to not need ejabberd-xmlrpc module anymore.
The Change Log is here. The eggs are on PyPi so you can update as usual. You can try the GUI of the beta 3.1 using this URL http://nightlies.archipelproject.org/beta3.1.
Enjoy and report any problem!
We will provide companies with professional support soon.