Re: [Duplicity-talk] Simple multi-generation backup

[edgar . soldin]

On 15.12.2022 14:35, Felix Natter wrote:
> hello Edgar,
>
> thanks again, answers are inline:
>
> On 13.12.22 15:46, edgar.soldin--- via Duplicity-talk wrote:
> > On 13.12.2022 15:21, Felix Natter wrote:
> > > hello Edgar,
> > >
> > > thanks for your reply, answers are inline:
> >
> > np Felix,
> >
> > see below
> >
> > > On 08.12.22 13:37, edgar.soldin--- via Duplicity-talk wrote:
> > > > hey Felix,
> > > >
> > > > On 08.12.2022 08:10, Felix Natter via Duplicity-talk wrote:
> > > > > Dear duplicity community,
> > > > >
> > > > > I would like to set up a _simple_ multi-generation backup using duplicity
> > > > > to a cloud service:
> > > > >
> > > > > - target $CLOUD/daily/: backup once per day, keep last X
> > > > > - target $CLOUD/weekly/: backup once per week, keep last Y
> > > > > - target $CLOUD/monthly/: backup once per month, keep last Z
> > > > >
> > > > > My daily script (to be called by cron job) looks like this:
> > > > >
> > > > > duplicity --encrypt-key KEY --full-if-older-than 5D /repos $CLOUD/daily/
> > > > > duplicity --encrypt-key KEY remove-all-but-n-full 2 $CLOUD/daily/
> > > > >
> > > > > And my weekly cron job would call this:
> > > > >
> > > > > duplicity --encrypt-key KEY --full-if-older-than 3W /repos $CLOUD/weekly/
> > > > > duplicity --encrypt-key KEY remove-all-but-n-full 2 $CLOUD/weekly/
> > > > >
> > > > > and so on for the monthly backup.
> > > > >
> > > > > Q1: Is that a valid/good way to do it?
> > > >
> > > > kind of wasteful, but sure. why are your fulls not located at the time frame's 
> > > > start?
> > >
> > > Sorry I do not understand. I do think that it is "wasteful" recomputing 
> > > weeklies/monthlies
> > > instead of moving some backups from daily to weekly or monthly ("rotation"),
> >
> > yeah, the doubled backup creation and file space requirement. you could just do 
> > weekly fulls with daily incrementals into a week specific target folder on the 
> > backend
> >
> > say you do per cron (eg. use date command to determine week of year `date 
> > +%Y.%U`)
> > do daily.. duplicity /repos $CLOUD/$(date +%%Y.%%U)/
> > NOTE: cron needs escaped % characters. escape by adding one % in front of the 
> > other ;)
> >
> > that way you will end up with weekly fulls with daily incrementals.
> >
> > don't forget to verify your backups regularly (to be safe).
> >
> > to keep only older weeklies, monthlies that you want you may delete weekly 
> > incrementals or just delete whole weekly folders as needed later.
>
> This sounds good, but our cloud storage does not have ssh access (only 
> sftp/scp),
> so I cannot automate deleting old weekly directories. 

sure you can automate scp/ftp deletion per shell script.

> As a workaround, I could use
>
> duplicity remove-older-than now --force $CLOUD/2022.50
> (repeat for all old week's folders)?

that will leave the folders and is actually not what it is meant for ;)

> Tests show that I cannot do this recursively, i.e.
>
> duplicity remove-older-than --force 5M $CLOUD
>
> won't find anything.

that is correct. duplicity will work with the one remote folder given and 
ignore any subfolders.

> > > but what do you mean by "fulls not located at the time frame's start"?
> >
> > weeks start at Monday. months start with the first of month.
>
> Ok, yes that is better.
>
> > > Is there a better way to do multi-generation backups with duplicity (or duply)?
> >
> > probably. anyway, see my suggestion above. you need to make sure the date is 
> > correct on the box all the time though ;)
>
> I think that is generally true for using duplicity with --time.

no what i mean is that in theory the box may loose the current date. say a 
power loss and empty bios battery. after that the local date will be wrong of 
course. it's a remote chance, but still something to consider maybe.

sunny regards.. ede/duply.net