Duplicity will nicht per Crontab funktionieren?!

pcace · Beitrag von **pcace** » 31.12.2018 00:18:11

Hi,

ich habe ien komisches Problem mit Duplicity: Ich nutze es um Serverbackups auf ein google-drive speicher zu schieben. Das funktioniert wunderbar, wenn ich das Script direkt ausführe, wenn ich es aber per crontab ausführe bekomme ich die fehlermeldung dass accounts.google.com nicht erreichbar sei. Hä?!?!? Wie kann das sein? ich weis überhaupt nicht wo ich anfangen soll zu suchen!

Log wenn ich das Backupscript per crontab ausführe:

Code: Alles auswählen

###### Starting backup on Mon Dec 31 00:00:01 CET 2018 ######

APT List  ...
Done

Dump Database ...
Creating backup for database mysql
Creating backup for database nextcloud
Creating backup for database vmail
Done

mainBackup....
Traceback (innermost last):
 File "/usr/bin/duplicity", line 1560, in <module>
   with_tempdir(main)
 File "/usr/bin/duplicity", line 1546, in with_tempdir
   fn()
 File "/usr/bin/duplicity", line 1385, in main
   action = commandline.ProcessCommandLine(sys.argv[1:])
 File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1135, in ProcessCommandLine
   backup, local_pathname = set_backend(args[0], args[1])
 File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1010, in set_backend
   globals.backend = backend.get_backend(bend)
 File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 223, in get_backend
   obj = get_backend_object(url_string)
 File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 209, in get_backend_object
   return factory(pu)
 File "/usr/lib/python2.7/dist-packages/duplicity/backends/pydrivebackend.py", line 69, in __init__
   gauth.CommandLineAuth()
 File "/usr/local/lib/python2.7/dist-packages/pydrive/auth.py", line 120, in _decorated
   self.Refresh()
 File "/usr/local/lib/python2.7/dist-packages/pydrive/auth.py", line 475, in Refresh
   self.credentials.refresh(self.http)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 545, in refresh
   self._refresh(http)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 761, in _refresh
   self._do_refresh_request(http)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 780, in _do_refresh_request
   body=body, headers=headers)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/transport.py", line 282, in request
   connection_type=connection_type)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1694, in request
   (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1434, in _request
   (response, content) = self._conn_request(conn, request_uri, method, body, headers)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1360, in _conn_request
   raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
ServerNotFoundError: Unable to find the server at accounts.google.com


cleanup alles ältere als 6 Monate
Traceback (innermost last):
 File "/usr/bin/duplicity", line 1560, in <module>
   with_tempdir(main)
 File "/usr/bin/duplicity", line 1546, in with_tempdir
   fn()
 File "/usr/bin/duplicity", line 1385, in main
   action = commandline.ProcessCommandLine(sys.argv[1:])
 File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1127, in ProcessCommandLine
   globals.backend = backend.get_backend(args[0])
 File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 223, in get_backend
   obj = get_backend_object(url_string)
 File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 209, in get_backend_object
   return factory(pu)
 File "/usr/lib/python2.7/dist-packages/duplicity/backends/pydrivebackend.py", line 69, in __init__
   gauth.CommandLineAuth()
 File "/usr/local/lib/python2.7/dist-packages/pydrive/auth.py", line 120, in _decorated
   self.Refresh()
 File "/usr/local/lib/python2.7/dist-packages/pydrive/auth.py", line 475, in Refresh
   self.credentials.refresh(self.http)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 545, in refresh
   self._refresh(http)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 761, in _refresh
   self._do_refresh_request(http)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 780, in _do_refresh_request
   body=body, headers=headers)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/transport.py", line 282, in request
   connection_type=connection_type)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1694, in request
   (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1434, in _request
   (response, content) = self._conn_request(conn, request_uri, method, body, headers)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1360, in _conn_request
   raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
ServerNotFoundError: Unable to find the server at accounts.google.com


3 cycles behalten...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Dec 29 18:47:31 2018
No old backup sets found, nothing deleted.
done
###### BACKUP DONE on Mon Dec 31 00:00:55 CET 2018 ######

Und so sieht es aus wenn ich es direkt ausführe:

Code: Alles auswählen

###### Starting backup on Mon Dec 31 00:13:03 CET 2018 ######

APT List  ...
Done

Dump Database ...
Creating backup for database mysql
Creating backup for database nextcloud
Creating backup for database vmail
Done

mainBackup....
Reading globbing filelist exclude
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Dec 29 18:47:31 2018
--------------[ Backup Statistics ]--------------
StartTime 1546211588.04 (Mon Dec 31 00:13:08 2018)
EndTime 1546211710.02 (Mon Dec 31 00:15:10 2018)
ElapsedTime 121.98 (2 minutes 1.98 seconds)
SourceFiles 39603
SourceFileSize 4944514868 (4.60 GB)
NewFiles 394
NewFileSize 4626927 (4.41 MB)
DeletedFiles 96
ChangedFiles 1127
ChangedFileSize 620962484 (592 MB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1617
RawDeltaSize 135881922 (130 MB)
TotalDestinationSizeChange 45697350 (43.6 MB)
Errors 0
-------------------------------------------------


cleanup alles ältere als 6 Monate
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Dec 29 18:47:31 2018
No old backup sets found, nothing deleted.

3 cycles behalten...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Dec 29 18:47:31 2018
No old backup sets found, nothing deleted.
done
###### BACKUP DONE on Mon Dec 31 00:15:27 CET 2018 ######

Wenn mich jemand in die richtige Richtung schubsen könnten wär ich sehr Dankbar!!

Gruß

EDIT: das hier ist das Backupscript:

Code: Alles auswählen

#!/bin/bash

LOG="/scripts/backup.log"
HOST='repo'
BK_FULL_FREQ="1M" # neues komplettes Backup alle....
BK_FULL_LIFE="6M" # älter als ... löschen
BK_KEEP_FULL="3"  # ... volle Backups behalten

cd /scripts/
##Logfdiles
exec > >(tee -i ${LOG})
exec 2>&1

echo "###### Starting backup on $(date) ######"
echo ''
echo "APT List  ..."

##Was ist installiert:
dpkg --get-selections > /root/backup/software.list

echo 'Done'
echo ''
echo 'Dump Database ...'
## Datenbanken Dump
/bin/bash /scripts/dbdump.sh
echo 'Done'
echo ''
cd /scripts/
export GOOGLE_DRIVE_SETTINGS=/scripts/credentials
echo 'mainBackup....'
duplicity \
        incremental \
        --no-encryption \
        --full-if-older-than $BK_FULL_FREQ \
        --exclude-filelist exclude \
        / \
        gdocs://username@gmail.com/Backup

echo ''
echo 'cleanup alles ältere als 6 Monate'

duplicity \
        remove-older-than $BK_FULL_LIFE --force \
        gdocs://username@gmail.com/Backup

echo ''
echo '3 cycles behalten...'
duplicity \
        remove-all-inc-of-but-n-full $BK_KEEP_FULL --force \
        gdocs://username@gmail.com/Backup

echo 'done'
echo "###### BACKUP DONE on $(date) ######"
mailx -a "From: „xxxx“ Backup <xxxx@xxxxxx.de>" -s "Backup | "Server xxxxx@xxxxxx.de < $LOG

cronoik · Beitrag von **cronoik** » 31.12.2018 00:31:13

Poste doch noch bitte deine crontab und teile uns bitte noch mit in welche crontab du etwas eingetragen hast.

pcace · Beitrag von **pcace** » 31.12.2018 00:33:15

Heyhey,

ich habe es in die root corntab eingetragen:

Code: Alles auswählen

@daily /bin/bash /scripts/Backup.sh

Gruß

cronoik · Beitrag von **cronoik** » 31.12.2018 00:49:33

Laeuft das Skript denn als root?

pcace · Beitrag von **pcace** » 31.12.2018 00:53:42

Wenn ich es manuell starte? Ja.

rendegast · Beitrag von **rendegast** » 31.12.2018 10:30:01

Code: Alles auswählen
@daily /bin/bash /scripts/Backup.sh

Vielleicht mal wegen des Environment so versuchen?

Code: Alles auswählen

@daily su - -c "/bin/bash /scripts/Backup.sh;"

Code: Alles auswählen

@daily su - -c "cd /pfad/arbeitsverzeichnis; /bin/bash /scripts/Backup.sh;"

pcace · Beitrag von **pcace** » 09.01.2019 19:55:51

Hey, ich habe das jetzt mal eine weile beobachtet. Meistens funktioniert es, manchmal nicht - mit einem "Timeout" fehler. Warum auch immer.
Wenn noch jemand empfehlungen hat, dann immer her damit

Nun aber noch eine andere Frage, parallel versuche ich hier im Lokalen Netz einen kleinen Netzwerkspeicher-Server auf ein NAS zu Backuppen - mit duplicity per Rsync.
Ein initiales Backup funktioniert:

Code: Alles auswählen

duplicity \
        incremental \
        --no-encryption \
        --asynchronous-upload \
        --no-compression \
        --full-if-older-than $BK_FULL_FREQ \
        --exclude-filelist exclude \
        / \
        rsync://timemachine:TMBackup@192.168.2.60:873:://TMBackup/HUGO_Backup/

Leider passiert wenn ich dann ein folgendes Backup mache folgendes:

Code: Alles auswählen

Last full backup date: Sat Jan  5 15:36:18 2019
Traceback (innermost last):
 File "/usr/local/bin/duplicity", line 1678, in <module>
   with_tempdir(main)
 File "/usr/local/bin/duplicity", line 1664, in with_tempdir
   fn()
 File "/usr/local/bin/duplicity", line 1510, in main
   do_backup(action)
 File "/usr/local/bin/duplicity", line 1646, in do_backup
   incremental_backup(sig_chain)
 File "/usr/local/bin/duplicity", line 664, in incremental_backup
   globals.backend)
 File "/usr/local/bin/duplicity", line 429, in write_multivol
   at_end = gpg.PlainWriteFile(tarblock_iter, tdp.name, globals.volsize)
 File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 476, in PlainWriteFile
   return GzipWriteFile(block_iter, filename, size, gzipped)
 File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 454, in GzipWriteFile
   new_block = next(block_iter)
 File "/usr/lib/python2.7/dist-packages/future/builtins/newnext.py", line 56, in newnext
   return iterator.__next__()
 File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 544, in __next__
   result = self.process(next(self.input_iter))  # pylint: disable=assignment-from-no-return
 File "/usr/lib/python2.7/dist-packages/future/builtins/newnext.py", line 59, in newnext
   return iterator.next()
 File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 210, in get_delta_iter
   for new_path, sig_path in collated:
 File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 307, in collate2iters
   relem2 = next(riter2)
 File "/usr/lib/python2.7/dist-packages/future/builtins/newnext.py", line 59, in newnext
   return iterator.next()
 File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 375, in combine_path_iters
   refresh_triple_list(triple_list)
 File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 362, in refresh_triple_list
   new_triple = get_triple(old_triple[1])
 File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 348, in get_triple
   path = next(path_iter_list[iter_index])
 File "/usr/lib/python2.7/dist-packages/future/builtins/newnext.py", line 59, in newnext
   return iterator.next()
 File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 257, in sigtar2path_iter
   for tarinfo in tf:
 File "/usr/lib/python2.7/tarfile.py", line 2510, in next
   tarinfo = self.tarfile.next()
 File "/usr/lib/python2.7/tarfile.py", line 2352, in next
   raise ReadError("unexpected end of data")
ReadError: unexpected end of data

Meine erste Vermutung war, dass irgendein Volume volläuft (denn das Backup ist 2,6TB groß), aber wenn ich über den Backupzeitraum watch -n 1 | df -h mir anschaue, läuft da nie nix voll...

Was kann das gerät meinen? unexpected end of data?

Hm, für jeden Hinweis bin ich dankbar!

cronoik · Beitrag von **cronoik** » 10.01.2019 02:15:56

pcace hat geschrieben:
09.01.2019 19:55:51
Hey, ich habe das jetzt mal eine weile beobachtet. Meistens funktioniert es, manchmal nicht - mit einem "Timeout" fehler. Warum auch immer.
Wenn noch jemand empfehlungen hat, dann immer her damit

Netzwerk nicht erreichbar? Hast du noch eine vollstaendige Fehlermeldung fuer uns? Eroeffne bitte fuer deine andere Frage einen gesonderten Thread.

Glur · Beitrag von **Glur** » 10.01.2019 07:08:35

Besteht die ursprüngliche Frage eigentlich noch oder ist das jetzt das Timeout Problem? Oder ist das Timeout-Problem jetzt für das lokale backup?

pcace · Beitrag von **pcace** » 12.01.2019 18:41:16

Hi,

ich habe einen neuen Thread für das neue Problem aufgemacht: viewtopic.php?f=32&t=171928
Zu dem ursprünglichen Problem:
Es funktioniert so mehr oder weniger 50/50. Wenn ich das Script manuell starte funktioniert es immer. Lasse ich das Script per crontab nachts laufen habe ich immer mal wieder folgende ausgabe:

Code: Alles auswählen

mainBackup....
Traceback (innermost last):
 File "/usr/bin/duplicity", line 1560, in <module>
   with_tempdir(main)
 File "/usr/bin/duplicity", line 1546, in with_tempdir
   fn()
 File "/usr/bin/duplicity", line 1385, in main
   action = commandline.ProcessCommandLine(sys.argv[1:])
 File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1135, in ProcessCommandLine
   backup, local_pathname = set_backend(args[0], args[1])
 File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1010, in set_backend
   globals.backend = backend.get_backend(bend)
 File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 223, in get_backend
   obj = get_backend_object(url_string)
 File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 209, in get_backend_object
   return factory(pu)
 File "/usr/lib/python2.7/dist-packages/duplicity/backends/pydrivebackend.py", line 69, in __init__
   gauth.CommandLineAuth()
 File "/usr/local/lib/python2.7/dist-packages/pydrive/auth.py", line 120, in _decorated
   self.Refresh()
 File "/usr/local/lib/python2.7/dist-packages/pydrive/auth.py", line 475, in Refresh
   self.credentials.refresh(self.http)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 545, in refresh
   self._refresh(http)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 761, in _refresh
   self._do_refresh_request(http)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 780, in _do_refresh_request
   body=body, headers=headers)
 File "/usr/local/lib/python2.7/dist-packages/oauth2client/transport.py", line 282, in request
   connection_type=connection_type)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1694, in request
   (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1434, in _request
   (response, content) = self._conn_request(conn, request_uri, method, body, headers)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1354, in _conn_request
   conn.connect()
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1100, in connect
   sock.connect(sockaddr)
 File "/usr/lib/python2.7/socket.py", line 228, in meth
   return getattr(self._sock,name)(*args)
timeout: timed out

Vllt hat ja noch jemand eine Idee was das sein könnte?

Gruß und Dank

debianforum.de

Duplicity will nicht per Crontab funktionieren?!

Duplicity will nicht per Crontab funktionieren?!

Re: Duplicity will nicht per Crontab funktionieren?!

Re: Duplicity will nicht per Crontab funktionieren?!

Re: Duplicity will nicht per Crontab funktionieren?!

Re: Duplicity will nicht per Crontab funktionieren?!

Re: Duplicity will nicht per Crontab funktionieren?!

Re: Duplicity will nicht per Crontab funktionieren?!

Re: Duplicity will nicht per Crontab funktionieren?!

Re: Duplicity will nicht per Crontab funktionieren?!

Re: Duplicity will nicht per Crontab funktionieren?!