Troubleshooting and Support Request

Troubleshooting

Check Openstack log files for error messages. Most of the errors related to storage are in Cinder or Nova logs.

Location of Openstack log files for Upstart and SysV based distros:

Project	Directory	Log Files
Cinder API	/var/log/cinder	cinder-api.log or api.log
Cinder Backup	/var/log/cinder	cinder-backup.log or backup.log
Cinder Manage	/var/log/cinder	cinder-manage.log or manage.log
Cinder Scheduler	/var/log/cinder	cinder-scheduler.log or scheduler.log
Cinder Volume	/var/log/cinder	cinder-volume.log or volume.log
Manila API	/var/log/manila	manila-api.log or api.log
Manila Manage	/var/log/manila	manila-manage.log or manage.log
Manila Scheduler	/var/log/manila	manila-scheduler.log or scheduler.log
Manila Share	/var/log/manila	manila-share.log or share.log
Manila Data	/var/log/manila	manila-data.log or data.log
Nova Compute	/var/log/nova	nova-compute.log

Shell command to check errors:

$ grep -r Traceback /var/log/{cinder,manila,nova}/*.log

Location of Openstack log files for systemd based distros:

Project	Service name (CentOS/Fedora/SUSE/RedHat)	Service name (Debian/Ubuntu)
Cinder API	openstack-cinder-api or cinder-api	cinder-api
Cinder Backup	openstack-cinder-backup or cinder-backup	cinder-backup
Cinder Manage	openstack-cinder-manage or cinder-manage	cinder-manage
Cinder Scheduler	openstack-cinder-scheduler or cinder-scheduler	cinder-scheduler
Cinder Volume	openstack-cinder-volume or cinder-volume	cinder-volume
Manila API	openstack-manila-api or manila-api	manila-api
Manila Manage	openstack-manila-manage or manila-manage	manila-manage
Manila Scheduler	openstack-manila-scheduler or manila-scheduler	manila-scheduler
Manila Share	openstack-manila-share or manila-share	manila-share
Manila Data	openstack-manila-data or manila-data	manila-data
Nova Compute	openstack-nova-compute or nova-compute	nova-compute

Shell command to check error:

$ journalctl -a -u service_name | grep Traceback

Note: the most effective way to find Python tracebacks from text is the tbgrep command-line tool:

$ sudo pip install tbgrep
$ tbgrep /var/log/{cinder,manila,nova}/*.log
$ journalctl -a -u service_name | tbgrep

If the error is not self explanatory, enable the debug logging, restart the service and try to reproduce the error. Debug loggings will trace all calls to Nexenta, which allows to narrow down the possible cause of the error.

To enable debug in cinder, add the following line to cinder.conf:

[DEFAULT]
debug = True

To enable debug in manila, add the following line to manila.conf:

[DEFAULT]
debug = True

And restart cinder volume or manila share service.

Upstart and SysV based distros: sudo service cinder-volume restart or sudo service manila-share restart

Systemd based distros: sudo systemctl restart cinder-volume or sudo systemctl restart openstack-cinder-volume, sudo systemctl restart manila-share or sudo systemctl restart openstack-manila-share

Support Request

When creating a support request please provide the following information:

NexentaStor version
Support bundle collected from NexentaStor
OpenStack version (e.g. Icehouse, Juno, Kilo, Liberty, Mitaka, Newton, Ocata, Pike, Queens, Rocky, Stein, Train, Ussuri, Victoria, Wallaby, Xena, Yoga or Zed)

Cinder or Manila driver version: go to appropriate driver location and copy-paste comments from the top of file:

Nexenta Version	Backend Type	Driver Location
NexentaStor4	Cinder NFS	/usr/lib/python2.7/dist-packages/cinder/volume/drivers/nexenta/nfs.py
NexentaStor4	Cinder iSCSI	/usr/lib/python2.7/dist-packages/cinder/volume/drivers/nexenta/iscsi.py
NexentaStor4	Manila NFS	/usr/lib/python2.7/dist-packages/manila/share/drivers/nexenta/ns4/nexenta_nas.py
NexentaStor5	Cinder NFS	/usr/lib/python2.7/dist-packages/cinder/volume/drivers/nexenta/ns5/nfs.py
NexentaStor5	Cinder iSCSI	/usr/lib/python2.7/dist-packages/cinder/volume/drivers/nexenta/ns5/iscsi.py
NexentaStor5	Manila NFS	/usr/lib/python2.7/dist-packages/manila/share/drivers/nexenta/ns5/nexenta_nas.py

Openstack service type (e.g. Cinder, Glance, Manila or Swift)
Collect OS version (e.g. Ubuntu 14.04, RHEL 7.0.x, CentOS 7.0.x) and OS specific information:
- cat cat /etc/lsb-release
- uname -a
- dpkg -l or rpm -qa
Collect copy of Cinder or Manila drivers folder
- /usr/lib/python2.7/dist-packages/cinder/volume/drivers/nexenta
- /usr/lib/python2.7/dist-packages/manila/share/drivers/nexenta
Collect Cinder or Manila configuration file:
- /etc/cinder/cinder.conf (default path)
- /etc/manila/manila.conf (default path)

Collect Nova, Cinder or Manila log files:

For Upstart and SysV based distros: /var/log/nova, /var/log/cinder or /var/log/manila

For Debian/Ubuntu and systemd:

for log in cinder-api cinder-backup cinder-manage cinder-scheduler cinder-volume \
    manila-api manila-manage manila-scheduler manila-share manila-data nova-compute; do 
    journalctl -a -u $log > $log.txt
done

For CentOS/Fedora/SUSE/RedHat and systemd:

for log in openstack-cinder-api openstack-cinder-backup openstack-cinder-manage \
    openstack-cinder-scheduler openstack-cinder-volume openstack-manila-api \
    openstack-manila-manage openstack-manila-scheduler openstack-manila-share \
    openstack-manila-data openstack-nova-compute; do
    journalctl -a -u $log > $log.txt
done

Collect log files and commands output from Nova Compute node(s):
- dmesg command output (possible system errors)
- mount command output (list of mounted filesystems)
- iscsiadm -m node -P1 command output (list of iSCSI targets)
- iscsiadm -m session -P3 command output (list of active iSCSI sessions and connections)
- iscsid service log file (depend on host OS and log settings: /var/log/messages file or journalctl | grep iscsid command output)
Steps to reproduce the issue, VM’s and volumes names, time frames, any custom scripts that customer ran, etc