Posts made by paco

@nikade Short answer is that the spikes seem to have smoothed out. I'm not really sure what happened. I don't have them at any resolution now. Perhaps some kind of heisenbug.

@olivierlambert Nothing dramatic. Just things "feeling slower than usual." That's qualititative. Just a gut hunch. So that's why I'm asking about ways to test or investigate.

@nikade You misunderstand. I am calling them "spikes", which most people use that word to mean things going dramatically up. Your graph shows spikes going up. Look closely at my graph, including the point I have highlighted. These "spikes" are 0. That is, it's humming along and then suddenly there's a spike down to 0. That's unlikely because the system is plenty active. So your graph shows spikes up, which is normal. Mine is showing occassional drops to 0, which is what I'm asking about.

In my throughput/IO graphs on Xen Orchestra, I'm seeing these spikes of 0 IOPS, 0 throughput, 0 latency, 0 IOWait. What I don't know is whether this is some kind of graphing artifact (i.e., it's not really happening, but the graph makes it look like it's happening), or whether I'm experiencing some kind of underlying hard disk throughput issue.

I'm in the middle of migrating a VM and it's estimating 12 hours to move 100G, which is long. I moved a couple 20G VMs this morning at 15-20 minutes each.

My question is, where could I look in XCP-ng logs to investigate? How could I get a sense of whether I'm really running into problems here? I've looked at these graphs many times over the years, and these spikes are new. This hasn't always looked like this.

Any thoughts?

The underlying storage is a LSI 9266-8i SAS RAID controller with 3 hard disks in a RAID5 array (hardware raid). I don't think I'm experiencing any other major issues, but maybe I am and don't know how to see it?

I'm running XCP-ng 8.2.1 and Xen Orchestra built from sources (well, from ronivay's Docker container) Xen Orchestra, commit e75cc Master, commit 70014.

@olivierlambert I appreciate the explanation. That suggests that there are no updates to 8.0.0? Or 8.1.0? 8.1.0 is 2 years old and EOL. I guess that's what EOL means: no more updates.

Not sure if this is expected behaviour or not. In the past, when I went to the Patches tab on my XCP-ng hosts, I would see patches listed here and I could choose to apply them. Now, I see none.

I have 2 hosts: one is running xcp-ng 8.0.0, the other is running 8.1.0. (They are single-host pools). In both cases, I see the green check mark and "host is up to date". There are no patches listed.

For the 8.0.0 host, I followed the upgrade from CLI instructions. The yum update command finished successfully and now it shows up in Xen Orchestra as running 8.2.1. But I was expecting Xen Orchestra to offer some amount of updating/patching/upgrading.

I haven't made any changes to the 8.1.0 host because I wanted to see if there was something I need to fix.

Do I misunderstand what I should see?

I'm running xo-server 5.89.0 and xo-web 5.94.0 from the sources via a Docker container.

Thanks,
Paco

@olivierlambert The proxy page has a pretty decent diagram for what things look like when there is no proxy. But it doesn't have a diagram for what things look like if there IS a proxy. I'm trying to understand it. Does this diagram capture the two situations?

If you're running 2 sites using a single instance of Xen Orchestra, is this what happens?

NodeJS is not my best language. But it looks _mount.js uses the execa library to wrap around standard child-process stuff.

I did a little bit of code reading and it looks like, ultimately, execa and child-process-promise end up invoking child_process.spawn(), and it seems like spawn() is just an asynchronous wrapper around child_process.exec(). This leads me to believe that the input ultimately ends up on a shell command line.

According to the docs I found:

child_process.exec(): spawns a shell and runs a command within that shell, passing the stdout and stderr to a callback function when complete

Now, both execa and the child-process-promise library do a lot of parsing of arguments. I can't tell to what extent that parsing and marshalling of command line arguments actually defeats injection attacks. I've done a lot of code review in my day, and sometimes I miss the line that does the work of defeating injection attacks because it is terse and effective. I might have looked right at it and misunderstood its importance.

Like I said, NodeJS and all the modules that are getting loaded in here to make this happen is difficult for me to follow. It's not my strong suit. Sorry if I'm wasting your time because I can't follow the code flow. It looks to me like it all boils down to exec(), and exec() by definition invokes the shell. But maybe we're protected by one of the argument marshalling functions in the call chain.

@julien-f Yeah, I have just found the useSudo option in the docs. I swear things change fast enough on xen orchestra that "upgrading" almost doesn't make sense. I rebuild it like every 3-6 months and so much changes that it's almost easier to start over. If I had started over and pretended I didn't know anything, I would have read the documentation and seen the useSudo option. That's not a complaint. Fast development and adding features is awesome.

I want to confirm a couple things about Xen Orchestra and NFS backups.

1. Running Mount from XOA

When I go to Settings -> Remotes and try to add an NFS remote, it looks like Xen Orchestra wants to run mount on the XOA VM to mount it. That implies that the web server is running as root or the web server has the ability to invoke commands as root, because only root can mount filesystems. That seems decidedly suboptimal from a security point of view. If I run xo-web as root and there's an input validation error, then I not only have command injection, but I have commands injected that are running as root.

Imagine someone types ver=3; echo blah | tee -a /etc/shadow; : or similar as their options on the NFS remote. That potentially executes as root and does something nasty (where blah is a correctly formatted /etc/shadow entry).

I could potentially create a rule in my /etc/sudoers file to allow the unprivileged www user to invoke /bin/mount as root with no password, but I'm still not sure how exposed I would be to command injection. The risk is limited to that one command, but it's still sorta there. We'd have to change the mount command that is executed to be sudo mount, instead of mount.

2. Dataflow

If I follow the flow, the hypervisor will transmit the VM's data for backup to my XOA VM, who in turn writes it out to the NFS volume. If my XOA instance is not on the same network or host, that's a bad idea, right? I mean: it would be better for the host to do the NFS mount from the hypervisor and write the data out to the NFS share directly, wouldn't it? Even if my XOA VM is on the same host, the data is transmitted into the VM and back out again. Do I understand this correctly?

@stormi It does. I put the screen shots of what it looks like when you run it in the pull request.

@stormi I understand. The change to the install.sh that I want to make is basically to detect FreeBSD and run the pkg install commands.

@stormi I had already forked, branched, and edited by the time I saw the shortcut. So it's here.

I'm not sure I understand why the code has to be compiled by the xcp-ng team, since that makes a pretty insurmountable hurdle. I can certainly walk someone through the process of doing that, and it can even be automated, but that's a lot of effort.

At the end of the day, when somone executes pkg install xe-guest-utilities they reach out to the official FreeBSD server and pull down a binary from there. So the security threat/impact of xcp-ng doing that is identical to the threat/impact of them doing that themselves.

There are ways to check the authenticity of the binary package rather than building it from source. That would let a Linux host fetch the file and verify it, rather than running enough FreeBSD hosts and automation to build two packages from source.

The documentation for Guest Tools says "The install.sh script doesn't support those systems, but there are ways to install the management agent anyway."

I'd be happy to contribute 2 things:

1. docs that cover how to install on FreeBSD (it's like one command)
2. a bit of shell script that detects FreeBSD and does the right FreeBSD-specific command to install the tools via the internet

Also, if you wanted to, you could probably pull in the binaries for FreeBSD's xe-guest-utilities and put them on the ISO. They're not big and they don't have dependencies.

Anyways, I can't find the source for install.sh so that I could write that bit of code. Which repo has the install.sh that goes into the guest tools ISO?

@Danp As soon as I read that, I thought "I think I've been down this road before." Sure enough. I gave it 2G of RAM and it wouldn't build. I gave it 4G of RAM and I watched using systat and that final step went to about 52% of RAM usage. It has completed. I routinely run Xen Orchestra on a 1G VM because it's rarely used and it runs fine that way. But building it clearly takes a lot more RAM. I'll update my internal notes. It might be worth it to update the building from source docs to mention having enough RAM.

Thanks for hitting the nail on the head with that advice!

I'm trying to build Xen Orchestra from source. The platform is:

• OS: FreeBSD 12.1-RELEASE
• node: 8.16.2
• yarn: 1.22.4

I am following the instructions. I checked out using git and did yarn and yarn build. Below is what I see. The real issue is that I'm just seeing 'Killed' and then an error.

I tried yarn --verbose build and that's the output that I see.

How can I get more debugging detail on the build process to find more details?

Using polyfills: No polyfills were added, since the `useBuiltIns` option was not set.
[11:30:29] Using gulpfile /opt/xen-orchestra/packages/xo-web/gulpfile.js
[11:30:29] Starting 'build'...
[11:30:29] Starting 'buildPages'...
[11:30:29] Starting 'buildScripts'...
[11:30:29] Starting 'buildStyles'...
[11:30:29] Starting 'copyAssets'...
Successfully compiled 22 files with Babel.
[11:30:38] Finished 'buildPages' after 9.63 s
[11:30:43] Finished 'buildStyles' after 14 s
[11:30:43] Finished 'copyAssets' after 14 s
Killed
* xo-web:build _ Error: null
_ 6

@olivierlambert You expect a system administrator to read documentation!?

Anyways, that was it. Sorry I didn't look more carefully. I rebuilt using node8 and poof it was running. Thanks guys.

@DustinB Brilliant. Thanks. I was using node 11. I know absolutely nothing about node versions, so I just went with the latest available. I'll try downgrading to node 8 and see how that goes.

I'm trying to build xen orchestra from source on FreeBSD. The build process runs to completion, but when I try to run the service (sudo yarn start) I get an error:

2019-05-01T19:27:54.132Z - xo:xo-server - [ERROR] uncaught exception
{ error:
   { Error: Could not locate the bindings file. Tried:

and then a long list of exceptions and stuff. I can provide it all if you like.

During the build I noticed an error scroll by quite fast. I though "it's probably not building correctly". So I finally isolated the package that isn't building.

I go to ~/xen-orchestra/node_modules/level-party and run 'yarn'. It seems like level-party builds its own version of leveldown. Now, at the higher level (~/xen-orchestra/node_modules/leveldown) the package has built fine with no errors. But when yarn tries to build ~/xen-orchestra/node_modules/level-party/node_modules/leveldown, I get this error:

 CXX(target) Release/obj.target/leveldown/src/batch.o
In file included from ../src/batch.cc:3:
In file included from ../../nan/nan.h:192:
../../nan/nan_maybe_43_inl.h:112:15: error: no member named 'ForceSet' in 'v8::Object'
  return obj->ForceSet(isolate->GetCurrentContext(), key, value, attribs);
/home/paco/.node-gyp/11.13.0/include/node/node.h:171:1: note: 'MakeCallback' has been explicitly marked deprecated here
NODE_DEPRECATED("Use MakeCallback(..., async_context)",
^
/home/paco/.node-gyp/11.13.0/include/node/node.h:91:20: note: expanded from macro 'NODE_DEPRECATED'
    __attribute__((deprecated(message))) declarator
                   ^

I'm assuming, perhaps wrongly, that the two things are related. I figure if I don't get a clean build, I shouldn't expect it to run.

I'm building master of https://github.com/vatesfr/xen-orchestra/. I'm on FreeBSD 12.0-RELEASEp3 on amd64.

Anybody have thoughts or things I could try?

Happy to paste more log files and more verbose errors, but these things are so large, that I didn't want to preemptively spew a ton of them into the forum.

Thanks!