XCP-ng 8.3 betas and RCs feedback đ
-
@stormi said in XCP-ng 8.3 beta :
@ph7 I did push a rebuilt netdata for Xen 4.17 yesterday, so it should have worked, but I admit I haven't tested the update process myself so maybe I've missed something.
Can you extract the error messages from the output of
yum history
?yum history
to get the list of operations, andyum history info NUMBER
for the details of a given call.[12:23 X3 ~]# yum history InlĂ€sta insticksmoduler: fastestmirror ID | Kommandorad | Datum och tid | Ă tgĂ€rd(er) | Ăndrade ------------------------------------------------------------------------------- 11 | update --disablerepo=* - | 2024-04-23 00:29 | I, U | 53 EE 10 | remove netdata-ui.x86_64 | 2024-04-23 00:26 | Erase | 2 9 | update --disablerepo=* - | 2024-04-18 18:11 | Update | 21 8 | update --disablerepo=* - | 2024-04-04 17:50 | Update | 1 7 | reinstall netdata.x86_64 | 2024-04-03 16:41 | Reinstall | 1 6 | reinstall netdata-ui | 2024-04-03 16:34 | Reinstall | 1 5 | --enablerepo=* install n | 2024-03-30 11:41 | Install | 1 4 | update --disablerepo=* - | 2024-03-29 14:13 | Update | 21 3 | install netdata-ui | 2024-03-23 13:59 | Install | 9 EE 2 | update --disablerepo=* - | 2024-03-01 16:00 | Update | 6 1 | update --disablerepo=* - | 2024-02-29 22:12 | I, UF, U | 63 history list --------------------------- [12:23 X3 ~]# yum history info NUMBER 10 InlĂ€sta insticksmoduler: fastestmirror Transaktions-ID: 10 Starttid : Tue Apr 23 00:26:27 2024 Start-rpmdb : 548:9dff1ffa674baaf209143925833d8346df54beac Sluttid : 00:27:27 2024 (60 sekunder) Slut-rpmdb : 546:17e105aef8586ef4615b3761b221b4051cc02275 AnvĂ€ndare : System <ej satt> Returkod : Lyckades Kommandoradsfel: remove netdata-ui.x86_64 netdata.x86_64 Transaktionen utförd med: Installerade rpm-4.11.3-32.el7.x86_64 @xcp:main/$releasever Installerade yum-3.4.3-158.el7.centos.noarch @xcp:main/$releasever Installerade yum-plugin-fastestmirror-1.1.31-46.el7_5.noarch @xcp:main/$releasever Ăndrade paket: Radering netdata-1.19.0-6.xcpng8.3.x86_64 @xcp-ng-base Radering netdata-ui-1.19.0-6.xcpng8.3.x86_64 @xcp-ng-base history info Excluding mirror: updates.xcp-ng.org * xcp-ng-updates: mirrors.xcp-ng.org Uppdaterade intel-ice-1.11.17.1-1.xcpng8.3.x86_64 @?xcp-ng-base Uppdatering 1.11.17.1-2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade intel-microcode-20231009-1.xcpng8.3.noarch @xcp-ng-base Uppdatering 20240130-1.xcpng8.3.noarch @xcp-ng-base Uppdaterade kernel-4.19.19-8.0.32.1.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 4.19.19-8.0.33.1.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade kexec-tools-1:2.0.15-18.xcpng8.3.x86_64 @?xcp-ng-base Uppdatering 1:2.0.15-19.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade libcgroup-0.41-15.el7.x86_64 @?xcp-ng-base Uppdatering 0.41-21.el7.x86_64 @xcp-ng-base Uppdaterade libcgroup-tools-0.41-15.el7.x86_64 @?xcp-ng-base Uppdatering 0.41-21.el7.x86_64 @xcp-ng-base Uppdaterade message-switch-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade python2-xapi-storage-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Ber-inst python3-scapy-2.4.5-3.xcpng8.3.noarch @xcp-ng-base Uppdaterade qemu-2:4.2.1-5.2.4.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 2:4.2.1-5.2.7.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade qlogic-fastlinq-8.55.13.0-1.xcpng8.3.x86_64 @?xcp-ng-base Uppdatering 8.74.0.2-1.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade rrdd-plugins-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade sm-3.0.12-6.2.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 3.0.12-12.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade sm-cli-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade squeezed-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade varstored-1.2.0-1.3.xcpng8.3.x86_64 @?xcp-ng-base Uppdatering 1.2.0-2.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade varstored-guard-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade varstored-tools-1.2.0-1.3.xcpng8.3.x86_64 @?xcp-ng-base Uppdatering 1.2.0-2.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade vcputune-2.0.1-5.xcpng8.3.noarch @?xcp-ng-base Uppdatering 2.0.2-1.xcpng8.3.noarch @xcp-ng-base Uppdaterade vhd-tool-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade vncterm-10.2.0-3.xcpng8.3.x86_64 @?xcp-ng-base Uppdatering 10.2.1-2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade wsproxy-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xapi-core-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xapi-nbd-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xapi-rrd2csv-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xapi-storage-script-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xapi-tests-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xapi-xe-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xcp-clipboardd-1.0.3-7.xcpng8.3.x86_64 @?xcp-ng-base Uppdatering 1.0.3-8.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xcp-featured-1.1.6-3.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 1.1.6-4.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xcp-networkd-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xcp-ng-release-8.3.0-18.x86_64 @xcp-ng-base Uppdatering 8.3.0-19.x86_64 @xcp-ng-base Uppdaterade xcp-ng-release-config-8.3.0-18.x86_64 @xcp-ng-base Uppdatering 8.3.0-19.x86_64 @xcp-ng-base Uppdaterade xcp-ng-release-presets-8.3.0-18.x86_64 @xcp-ng-base Uppdatering 8.3.0-19.x86_64 @xcp-ng-base Uppdaterade xcp-rrdd-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xen-crashdump-analyser-2.5.6-1.xcpng8.3.x86_64 @?xcp-ng-base Uppdatering 2.6.1-1.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xen-dom0-libs-4.13.5-10.58.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 4.17.3-4.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xen-dom0-tools-4.13.5-10.58.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 4.17.3-4.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xen-hypervisor-4.13.5-10.58.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 4.17.3-4.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xen-libs-4.13.5-10.58.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 4.17.3-4.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xen-tools-4.13.5-10.58.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 4.17.3-4.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xenopsd-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xenopsd-cli-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xenopsd-xc-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade xenserver-status-report-1.3.14-1.xcpng8.3.noarch @xcp-ng-base Uppdatering 2.0.3-1.xcpng8.3.noarch @xcp-ng-base Skriptutdata: 1 warning: /etc/sysconfig/xencommons saved as /etc/sysconfig/xencommons.rpmsave history info -------------------------- [12:25 X3 ~]# yum history info NUMBER 11 InlĂ€sta insticksmoduler: fastestmirror Transaktions-ID: 11 Starttid : Tue Apr 23 00:29:47 2024 Start-rpmdb : 546:17e105aef8586ef4615b3761b221b4051cc02275 Sluttid : 00:31:19 2024 (92 sekunder) Slut-rpmdb : 547:ab22718930a5c79c4616bc82131af44e4ed6cb50 AnvĂ€ndare : System <ej satt> Returkod : Lyckades Kommandoradsfel: update --disablerepo=* --enablerepo=xcp-ng-base,xcp-ng-updates -y Transaktionen utförd med: Installerade rpm-4.11.3-32.el7.x86_64 @xcp:main/$releasever Installerade yum-3.4.3-158.el7.centos.noarch @xcp:main/$releasever Installerade yum-plugin-fastestmirror-1.1.31-46.el7_5.noarch @xcp:main/$releasever Ăndrade paket: Uppdaterade blktap-3.54.6-2.1.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 3.54.6-3.1.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade edk2-20220801-1.7.3.2.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 20220801-1.7.4.1.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade forkexecd-23.31.0-1.7.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 24.11.0-1.2.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade gpumon-0.25.0-6.1.xcpng8.3.x86_64 @xcp-ng-base Uppdatering 0.25.0-14.1.xcpng8.3.x86_64 @xcp-ng-base Uppdaterade guest-templates-json-2.0.6-1.1.xcpng8.3.noarch @xcp-ng-base Uppdatering 2.0.9-1.1.xcpng8.3.noarch @xcp-ng-base Uppdaterade guest-templates-json-data-linux-2.0.6-1.1.xcpng8.3.noarch @xcp-ng-base Uppdatering 2.0.9-1.1.xcpng8.3.noarch @xcp-ng-base Uppdaterade guest-templates-json-data-other-2.0.6-1.1.xcpng8.3.noarch @xcp-ng-base Uppdatering 2.0.9-1.1.xcpng8.3.noarch @xcp-ng-base Uppdaterade guest-templates-json-data-windows-2.0.6-1.1.xcpng8.3.noarch @xcp-ng-base Uppdatering 2.0.9-1.1.xcpng8.3.noarch @xcp-ng-base Loading mirror speeds from cached hostfile Excluding mirror: updates.xcp-ng.org * xcp-ng-base: mirrors.xcp-ng.org --------------------------- I also got the result from my update when I discovered net netdata issue: yum update --disablerepo=* --enablerepo=xcp-ng-base,xcp-ng-updates -y ....... Fel: Paket: netdata-1.19.0-6.xcpng8.3.x86_64 (@xcp-ng-base) Behöver: libxenlight.so.4.13()(64bit) Tar bort: xen-dom0-libs-4.13.5-10.58.xcpng8.3.x86_64 (@xcp-ng-base) libxenlight.so.4.13()(64bit) Uppdaterat av: xen-dom0-libs-4.17.3-4.xcpng8.3.x86_64 (xcp-ng-base) ~libxenlight.so.4.17()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.4-10.27.xcpng8.3.x86_64 (xcp-ng-base) libxenlight.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.4-10.35.xcpng8.3.x86_64 (xcp-ng-base) libxenlight.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.4-10.36.xcpng8.3.x86_64 (xcp-ng-base) libxenlight.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.5-10.40.xcpng8.3.x86_64 (xcp-ng-base) libxenlight.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.5-10.42.xcpng8.3.x86_64 (xcp-ng-base) libxenlight.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.5-10.42.1.xcpng8.3.x86_64 (xcp-ng-base) libxenlight.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.5-10.42.3.xcpng8.3.x86_64 (xcp-ng-base) libxenlight.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.5-10.51.xcpng8.3.x86_64 (xcp-ng-base) libxenlight.so.4.13()(64bit) Fel: Paket: netdata-1.19.0-6.xcpng8.3.x86_64 (@xcp-ng-base) Behöver: libxenstat.so.4.13()(64bit) Tar bort: xen-dom0-libs-4.13.5-10.58.xcpng8.3.x86_64 (@xcp-ng-base) libxenstat.so.4.13()(64bit) Uppdaterat av: xen-dom0-libs-4.17.3-4.xcpng8.3.x86_64 (xcp-ng-base) ~libxenstat.so.4.17()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.4-10.27.xcpng8.3.x86_64 (xcp-ng-base) libxenstat.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.4-10.35.xcpng8.3.x86_64 (xcp-ng-base) libxenstat.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.4-10.36.xcpng8.3.x86_64 (xcp-ng-base) libxenstat.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.5-10.40.xcpng8.3.x86_64 (xcp-ng-base) libxenstat.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.5-10.42.xcpng8.3.x86_64 (xcp-ng-base) libxenstat.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.5-10.42.1.xcpng8.3.x86_64 (xcp-ng-base) libxenstat.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.5-10.42.3.xcpng8.3.x86_64 (xcp-ng-base) libxenstat.so.4.13()(64bit) TillgĂ€ngliga: xen-dom0-libs-4.13.5-10.51.xcpng8.3.x86_64 (xcp-ng-base) libxenstat.so.4.13()(64bit) Du kan försöka anvĂ€nda --skip-broken för att gĂ„ runt problemet Du kan försöka köra: rpm -Va --nofiles --nodigest
Sorry for the Swedish languish and the long text
Do You need a translation ???? -
I edited your post to correctly use Markdown code syntax.
-
@ph7 It didn't see
netdata-1.19.0-7.xcpng8.3.x86_64
was available. Probably you got the repo metadata in the time between when I published all the packages but netdata, and the time when I published netdata, and then yum didn't refresh the metadata because they were "fresh enough" in its eyes.Installing
netdata
should work now. -
@stormi
I was hot yesterday evening
maybe i should wait a few minutes at the next updateI installed
netdata-1.19.0-7.xcpng8.3.x86_64
netdata-ui-1.19.0-7.xcpng8.3.x86_64
Can confirm that it works fine direct in the browser.
Not tested inside xoAnd as you can see, I have now taken a quick course in markdown for dummies
-
after last bunch of updates via XO and toolstack restart, maintenance mode not disabled automaticaly.
going fine after reboot. -
@stormi said in XCP-ng 8.3 beta :
@xerxist said in XCP-ng 8.3 beta :
Which kernel are you looking at since 4.19 will be EOL in 9 months?
So, the main blocker in the way to upgrade the kernel is a kernel module we use for storage access from the VMs.
I'm curious: which module?
-
A couple of issues in my environment:
-
Stats are still broken for me, even after ensuring XO was updated (from source) and that the latest patches were applied to my pool. Was there a public GH issue tracking this breakage? (I hope to use the issue to track back to the code to find out why and see if there's something unique about my environment that might be the cause)
-
I know pure PV VMs are entirely unsupported, so I don't expect this to go anywhere, so no big deal if this remains broken (for me anyways!) but after the latest updates XAPI throws "shadow_allocation_set ${sz}MB invalid argument" (where ${sz} depends on static memory config for that VM) when I try to start a PV VM. To the best of my understanding, shadow pages are only supported for HVM, right?
Thanks!
Kevin -
-
@r0ssar00 You should convert PV VMs to HVM. However, if you really can't, you can switch them from
pv
topv-in-pvh
. It's not documented yet but this blog post will give you an example: https://xcp-ng.org/blog/2022/01/17/removing-support-for-32-bit-pv-guests/ -
@r0ssar00 Regarding broken stats, you could try to find logs related to them. Failures related to "stat" or "rrd" keywords, possibly. There was no github issue opened, so most details about the issue others had will only be found in this thread.
-
@r0ssar00 Could this be your issue with missing stats?
If statistics (all VMs and hosts) are not showing for a specific pool, check if there is a Backup network configured on your pool (setting is in the Advanced tab of the pool) and make sure XO can access all hosts of the pool via this network.
-
@Danp this was exactly it, thanks! Although I'm 100% sure the network is accessible to both hosts (they're directly connected to each other, no switch in between, ping to/from hosts/VMs-on-either-host works just fine), clearing it resolved the problem. I tried the other two networks I have set up (one is directly connected, one goes through a switch and is shared with the rest of my LAN) and only using the primary management interface worked (which kinda defeats the purpose...).
If this strikes you as a bug, happy to collect logs/etc, just let me know what and (a pointer to) how!
-
@stormi My inner "I want to do something slightly off the beaten path" nerd will be disappointed, but that's really all this was, just trying something different to be different for the challenge. Just thought I'd note it since it's not outside the realm of possible (in my mind anyways) that this is a symptom of potentially incomplete code ("failing to handle or improperly handling a config value" -> "is this the only place that happened?")
-
I have a suggestion w.r.t interface-rename.
Can we have a screen at the installation (using iso method) which lists all the NICs and we can rename the interfaces at install time itself ? -
-
I just pushed several updates to the
xcp-ng-base
repository for XCP-ng 8.3.Most notable changes
- Addition of a new SR driver,
largeblock
, which emulates 512B blocks for disks which only support 4KiB blocks. More at: https://xcp-ng.org/forum/topic/8901/largeblocksr-for-4kib-blocksize-disks - XO Lite updated to version 0.2.2, which changes the treeview component.
- new API for PCI passthrough, that Xen Orchestra already knows how to use to let you configure a device for passthrough.
- changes in the way configuration files are handled on OpenSSH. Previously, any update to
sshd_config
orssh_config
would overwrite any change you made to the files. This is what XenServer 8 does, but we wanted to do it differently, which is now the case. So, now, the update won't touch the files if you had modified them. To update default cipher lists, keys, algorithms, we now do so directly at build time, in the binaries. You can check this withsshd -T
. If you do have local changes to these files, the update won't be able to remove the cipher lists and other definitions from the files. Instead,name_of_file.rpmnew
files will be created in the/etc/ssh
directory. Which means next time we update the default lists, your configuration file will still override them with the old settings. Make sure to remove the lines starting withCiphers
,MACs
,KexAlgorithms
andHostKeyAlgorithms
if they are present. - changes in the way UEFI certificates are propagated to UEFI VMs at their first boot. In a few words: if you enabled Secure Boot for a VM but haven't setup your pool for Secure Boot first by running
secureboot-certs install
, then these VMs are currently in UEFI setup mode, which discards any Secure Boot setting. With the change made, Secure Boot will fail instead in this situation, thus avoiding a false sense of security. Future UI improvements in Xen Orchestra will help detecting the issue and fixing the UEFI certificates so that Secure Boot works in these VMs. You can also do it manually with:varstore-sb-state $VM_UUID user
(yes,user
, verbatim. It stands foruser mode
). This will install the necessary certificates to the VM provided you first have setup your pool for SecureBoot withsecureboot-certs install
.
- Addition of a new SR driver,
-
@stormi
I can not see anything underNetwork throughput
in XO-liteedit:
When I hover I can see the numbers but no graph -
@ph7
With another xcp-ng server that is not yet updated i can see theNetwork throughput
graph fromurl/xolite.html
which is onv0.2.2 (3a44b)
-
@gduperrey can you check?
-
@stormi Nice work on the PCI passthru. Very helpful. I like the treeview as well. Thanks!
-
-
@gduperrey
Both servers was last updated ~ 3 weeks agoI updated the other server the same way i updated the first one and its showing the graph
The first one still no graph after another reboot.
Netdata and XO is showing the graph by the way.Is there some log I can check
I have earlier ( a few month ago) had problems with no graphs shown at all but a change from dark to light mode and back to dark made them visible. But I dont remember what version it was and I have changed server after that.