Pkg slides from BSDCan conference

FreeBSD package management system

Vsevolod [email protected]

BSDCan May 17, 2014

[email protected]

What is pkg

Pkg (previously pkgng) is the binary package managementsystem written for FreeBSD.

I Binary packages management

I Replaces old pkg_* tools

I Uses central sqlite3 based storage

I Provides the comprehensive toolset for binary packagesmanagement

Vsevolod Stakhov [email protected] 2 of 26

[email protected]

Pkg development goals

The main goal of pkg is to simplify systemmanagement tasks.

I Easy install, remove and upgrade of binary packages

I Integration with the ports

I Automatic resolving of dependencies and conflicts

I Provide secure package management tool

I Encourage users to install software from binary packages

I . . . but do not prevent users from building custompackages using the ports


[email protected]

Planned ports and pkg interaction

Build STAGE Packages Pkg Instal

Build Depends

Run Depends


[email protected]

What is new in pkg 1.3

I New solver that can automatically resolve complexupgrade or install scenarios

I Improved security by sandboxing untrusted operations:

Pkg environment Sandbox

Unsafe operation

Result

Sandboxing:I archives extractingI vulnxml parsingI repositories signatures checking and public keys

extracting

I Concurrent locking system


[email protected]

Pkg architecture

Mirrors Jobsfetch

Solver

Checksum

+ Manifest

Files...

Package Install

FS

PkgDB

User

Request

Remote

Repos


[email protected]

The problems of the old solver in pkg

I Absence of conflicts resolving

I No alternatives support (plain dependencies only)

I Can perform merely a single task: either install orupgrade or remove


[email protected]

Tasks to solve

I Ports renaming:I simple: racket-textual→ racket-minimalI splitting/merging:

A

A B C A B C

A

I Ports reorganising:I files movingI dependencies changeI adding or removing new conflicts


[email protected]

Tasks to solve

There are another issues to be resolved:

I Find conflicts using files listI Set jobs priorities using the following rules:

I install dependencies firstI check for reverse dependencies and increase priorityI deal with conflicts using the same priorityI packages removing reverses the priority order


[email protected]

Existing systems

There are many examples of solvers used in different packagemanagement systems, for example:

I Zypper/SUSE - uses libsolv as the base

I Yum/RedHat - migrating to libsolv

I OpenBSD/pkg_add - uses internal solver

I Apt/Debian - uses internal solver

I Pacman/Archlinux - uses internal solver


[email protected]

External solvers

To interact with an external solver we have chosen the CUDFformat used in the Mancoosi research projecthttp://mancoosi.org:

package: devel/libblahversion: 1depends: x11/libfoo

package: security/blahversion: 2depends: devel/libblahconflicts: security/blah-devel


http://mancoosi.org

[email protected]

Interaction with external solver

There are some limitations and incompatibilities with CUDF.

I CUDF supports plain integers as versions and we need toconvert versions twice

I There is no support of options in CUDF packages formulas

I External solvers are often too complicated and large

I CUDF transformation is expensive in terms ofperformance


[email protected]

We need an internal solver!

Alternatives:

I Write own logic of dependencies and conflicts resolution?

I Use some existing solution?

I Use some known algorithm?

Use SAT solver for packages management

SAT expression︷︸︸︷(x1‖¬x2‖x3)︸︷︷︸

Clause

&(x3‖¬x1)&(x2)


[email protected]

Packages universeWe convert all packages involved to a packages universe ofthe following structure:

Name

Version

Con�icts Chain

Depend

Con�ict

N1 N2 N3 Nn

V1 V1 V1 V1

V2 V2 V2

V3


[email protected]

Making a SAT problem

I Assign a variable to each package: package A→ a1,package B→ b1

I Interpret a request as a set of unary clauses:I Install/Upgrade package A→ (a1)I Delete package B→ (¬b1)

I Convert dependencies and conflicts to disjuncted clauses


[email protected]

Converting dependencies and conflicts

I If package A depends on package B (versions B1 and B2),then we can either have package A not installed or any ofB installed:

(¬A‖B1‖B2)

I If we have a conflict between versions of B (B1, B2 and B3)then we ensure that merely one version is installed:

(¬B1‖¬B2)&(¬B1‖¬B3)&(¬B2‖¬B3)︸︷︷︸Conflicts chain


[email protected]

The solving of SAT problem

Some rules to follow to speed up SAT problem solving.

I Trivial propagation - solve unary clauses

I Unit propagation - solve clauses with only a singleunsolved variable

I DPLL algorithm backtracking.

I Package specific assumptions.


[email protected]

SAT problem propagation

I Trivial propagation - direct install or delete rules

(¬A‖B)& (A)︸︷︷︸true

&(¬C)︸︷︷︸false

&(¬A‖¬D)

I Unit propagation - simple depends and conflicts

Dependency

(¬A‖B)︸︷︷︸B→true

&

true︷︸︸︷(A) &

false︷︸︸︷(¬C)&

Conflict

(¬A‖¬D)︸︷︷︸D→false


[email protected]

DPLL algorithm

DPLL is proved to be one of the efficient algorithms to solveSAT problem (not the fastest but more simple thanalternatives).

x1

x2

x3

1

0

1


[email protected]

DPLL algorithm


x1

x2

x3

1

0

1

0


[email protected]

DPLL algorithm


x1

x2

x3

1

0

1

0

1

0 0

1


[email protected]

Package specific assumptions

Pure SAT solvers cannot deal with package management asthey do not consider several packages peculiarities:

I try to keep installed packages (if no direct conflicts)

I do not install packages if they are not needed (but try toupgrade if a user has requested upgrade)

These options also improve SAT performance providing agood initial assignment.


[email protected]

Solvers and Pkg

I Pkg may pass the formed universe to an external CUDFsolver:

I convert versionsI format requestI parse output

I Alternatively the internal SAT solver may be used:I convert the universe to SAT problemI formulate requestI ???I PROFIT


[email protected]

Perspectives

I Using pkg solver for ports management

I Better support of multiple repositories

I Test different solvers algorithms using CUDF

I New dependencies and conflicts format

I Provides and alternatives


[email protected]

New dependencies format

libblah >= 1.0+ option1,+option2‖libfoo! = 1.1

I Can depend on normal packages and virtualpackages (provides)

I Easy to define the concrete dependency versionsI Alternative dependencies

P1Depends

P2 Vx>

Vy Vz !

P3

Or

Con ict


[email protected]

Alternatives

I Used to organize packages with the same functionality(e.g. web-browser)

I May be used to implement virtual dependencies(provides/requires)

Alternatives

Text EditorDepends

Package


[email protected]

Existing issues to be solved prior to 1.3

I A solver cannot find install candidates for non-automatictop level packages (those without reverse depends)

I Package upgrade is performed improperly (need torename, install and unlink)

I Minor issues and crashes


[email protected]

Questions?

[email protected]


[email protected]

[email protected]

Pkg slides from BSDCan conference

Software

Transcript of Pkg slides from BSDCan conference