PostgreSQL La base de donnees la plus sophistiquee au monde.

La planète francophone de PostgreSQL

lundi 27 juillet 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 26 juillet 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juillet

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150726223049.GB16677@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Heikki Linnakangas pushed:

  • Handle AT_ReAddComment in test_ddl_deparse, and add a catch-all default. In the passing, also move AT_ReAddComment to more logical position in the enum, after all the Constraint-related subcommands. This fixes a compiler warning, added by commit e42375fc. Backpatch to 9.5, like that patch. http://git.postgresql.org/pg/commitdiff/13f2db2ffb2fac24fcb57ecc56e030e1145df127
  • Sanity-check that a page zeroed by redo routine is marked with WILL_INIT. There was already a sanity-check in the other direction: if a page was marked with WILL_INIT, it had to be initialized by the redo routine. It's not strictly necessary for correctness that a page is marked with WILL_INIT if it's going to be initialized at redo, but it's a missed optimization if nothing else. Fix a few instances of this issue in SP-GiST, where a block in WAL record was not marked with WILL_INIT, but was in fact always initialized at redo. We were creating a full-page image of the page unnecessarily in those cases. Backpatch to 9.5, where the new WILL_INIT flag was added. http://git.postgresql.org/pg/commitdiff/eb11de8ff5eac3592d539ad7ca3059c02e4d3e99
  • Add selectivity estimation functions for intarray operators. Uriy Zhuravlev and Alexander Korotkov, reviewed by Jeff Janes, some cleanup by me. http://git.postgresql.org/pg/commitdiff/c6fbe6d6fb828f50b9d67627588eb5ab8bd25e47
  • Fix off-by-one error in calculating subtrans/multixact truncation point. If there were no subtransactions (or multixacts) active, we would calculate the oldestxid == next xid. That's correct, but if next XID happens to be on the next pg_subtrans (pg_multixact) page, the page does not exist yet, and SimpleLruTruncate will produce an "apparent wraparound" warning. The warning is harmless in this case, but looks very alarming to users. Backpatch to all supported versions. Patch and analysis by Thomas Munro. http://git.postgresql.org/pg/commitdiff/766dcfb16ca385274d510eaed01724bb3836efdd

Ãlvaro Herrera pushed:

Teodor Sigaev pushed:

Andrew Dunstan pushed:

  • Fix location of output logs of pg_regress. initdb.log and postmaster.log were moved to within the temporary instance path by commit dcae5fa. This directory now gets removed at the end of the run of pg_regress when there are no failures found, which makes analysis of after-run issues difficult in some cases, and reduces the output verbosity of the buildfarm after a run. Fix by Michael Paquier Backpatch to 9.5 http://git.postgresql.org/pg/commitdiff/9faa6ae14f6098e4b55f0131f7ec2694a381fb87
  • Redirect install output of make check into a log file. dbf2ec1a changed make check so that the installation logs get directed to stdout and stderr. Per discussion on -hackers, this patch restores saving it to a file. It is now saved in /tmp_install/log, which is created once per invocation of any make target doing regression tests. Along the way, add a missing /log/ entry to test_ddl_deparse's .gitignore. Michael Paquier. http://git.postgresql.org/pg/commitdiff/16c33c50e122e3e7d03fc7ddd5cbd105c0118234
  • Fix treatment of nulls in jsonb_agg and jsonb_object_agg. The wrong is_null flag was being passed to datum_to_json. Also, null object key values are not permitted, and this was not being checked for. Add regression tests covering these cases, and also add those tests to the json set, even though it was doing the right thing. Fixes bug #13514, initially diagnosed by Tom Lane. http://git.postgresql.org/pg/commitdiff/d9a356ff2e6bb7ed5fb1145af49fa3e51e68a98a
  • Restore use of zlib default compression in pg_dump directory mode. This was broken by commit 0e7e355f27302b62af3e1add93853ccd45678443 and friends, which ignored the fact that gzopen() will treat "-1" in the mode argument as an invalid character, which it ignores, and a flag for compression level 1. Now, when this value is encountered no compression level flag is passed to gzopen, leaving it to use the zlib default. Also, enforce the documented allowed range for pg_dump's -Z option, namely 0 .. 9, and remove some consequently dead code from pg_backup_tar.c. Problem reported by Marc Mamin. Backpatch to 9.1, like the patch that introduced the bug. http://git.postgresql.org/pg/commitdiff/caef94d59fcfa1087be36d4a8b5ed4523872bf55

Tom Lane pushed:

  • Fix some oversights in BRIN patch. Remove HeapScanDescData.rs_initblock, which wasn't being used for anything in the final version of the patch. Fix IndexBuildHeapScan so that it supports syncscan again; the patch broke synchronous scanning for index builds by forcing rs_startblk to zero even when the caller did not care about that and had asked for syncscan. Add some commentary and usage defenses to heap_setscanlimits(). Fix heapam so that asking for rs_numblocks == 0 does what you would reasonably expect. As coded it amounted to requesting a whole-table scan, because those "--x <= 0" tests on an unsigned variable would behave surprisingly. http://git.postgresql.org/pg/commitdiff/434873806a9b1c0edd53c2a9df7c93a8ba021147
  • Fix add_rte_to_flat_rtable() for recent feature additions. The TABLESAMPLE and row security patches each overlooked this function, though their errors of omission were opposite: RLS failed to zero out the securityQuals field, leading to wasteful copying of useless expression trees in finished plans, while TABLESAMPLE neglected to add a comment saying that it intentionally *isn't* deleting the tablesample subtree. There probably should be a similar comment about ctename, too. Back-patch as appropriate. http://git.postgresql.org/pg/commitdiff/46d0a9bfac3d5221702318cc1cf119221d729c84
  • Redesign tablesample method API, and do extensive code review. The original implementation of TABLESAMPLE modeled the tablesample method API on index access methods, which wasn't a good choice because, without specialized DDL commands, there's no way to build an extension that can implement a TSM. (Raw inserts into system catalogs are not an acceptable thing to do, because we can't undo them during DROP EXTENSION, nor will pg_upgrade behave sanely.) Instead adopt an API more like procedural language handlers or foreign data wrappers, wherein the only SQL-level support object needed is a single handler function identified by having a special return type. This lets us get rid of the supporting catalog altogether, so that no custom DDL support is needed for the feature. Adjust the API so that it can support non-constant tablesample arguments (the original coding assumed we could evaluate the argument expressions at ExecInitSampleScan time, which is undesirable even if it weren't outright unsafe), and discourage sampling methods from looking at invisible tuples. Make sure that the BERNOULLI and SYSTEM methods are genuinely repeatable within and across queries, as required by the SQL standard, and deal more honestly with methods that can't support that requirement. Make a full code-review pass over the tablesample additions, and fix assorted bugs, omissions, infelicities, and cosmetic issues (such as failure to put the added code stanzas in a consistent ordering). Improve EXPLAIN's output of tablesample plans, too. Back-patch to 9.5 so that we don't have to support the original API in production. http://git.postgresql.org/pg/commitdiff/dd7a8f66ed278eef2f001a98e2312336c61ee527
  • Update oidjoins regression test for 9.5. New FK relationships for pg_transform. Also findoidjoins now detects a few relationships it didn't before for pre-existing catalogs, as a result of new regression tests leaving entries in those catalogs that weren't there before. http://git.postgresql.org/pg/commitdiff/158d61534e98638106d85bdb1de5dbdb56bc8057
  • In pg_ctl, report unexpected failure to stat() the postmaster.pid file. Any error other than ENOENT is a bit suspicious here, and perhaps should not be grounds for assuming the postmaster has failed. For the moment though, just report it, and don't change the behavior otherwise. The intent is mainly to try to determine why we are seeing intermittent failures in this area on some buildfarm members. Back-patch to 9.5 where some of these failures have happened. http://git.postgresql.org/pg/commitdiff/b7b5a1899aa3caeef30117f8e36c1f0e68e8847a
  • Some platforms now need contrib/tsm_system_time to be linked with libm. Buildfarm member hornet, at least, seems to want -lm in the link command. Probably this is due to the just-added use of isnan(). http://git.postgresql.org/pg/commitdiff/c879d51c5918ab5fc8feb9624aa4eae10ee93094
  • Dodge portability issue (apparent compiler bug) in new tablesample code. Some of the older OS X critters in the buildfarm are failing regression, with symptoms showing that a request for 100% sampling in BERNOULLI or SYSTEM methods actually gets only around 50% of the table. gdb revealed that the computation of the "cutoff" number was producing 0x7FFFFFFF rather than the expected 0x100000000. Inspecting the assembly code, it looks like gcc is trying to use lrint() instead of rint() and then fumbling the conversion from long double to uint64. This seems like a clear compiler bug, but assigning the intermediate result into a plain double variable works around it, so let's just do that. (Another idea would be to give up one bit of hash width so that we don't need to use a uint64 cutoff, but let's see if this is enough.) http://git.postgresql.org/pg/commitdiff/d9476b83808a39d9985845071bf0a150a3063b37
  • Make entirely-dummy appendrels get marked as such in set_append_rel_size. The planner generally expects that the estimated rowcount of any relation is at least one row, *unless* it has been proven empty by constraint exclusion or similar mechanisms, which is marked by installing a dummy path as the rel's cheapest path (cf. IS_DUMMY_REL). When I split up allpaths.c's processing of base rels into separate set_base_rel_sizes and set_base_rel_pathlists steps, the intention was that dummy rels would get marked as such during the "set size" step; this is what justifies an Assert in indxpath.c's get_loop_count that other relations should either be dummy or have positive rowcount. Unfortunately I didn't get that quite right for append relations: if all the child rels have been proven empty then set_append_rel_size would come up with a rowcount of zero, which is correct, but it didn't then do set_dummy_rel_pathlist. (We would have ended up with the right state after set_append_rel_pathlist, but that's too late, if we generate indexpaths for some other rel first.) In addition to fixing the actual bug, I installed an Assert enforcing this convention in set_rel_size; that then allows simplification of a couple of now-redundant tests for zero rowcount in set_append_rel_size. Also, to cover the possibility that third-party FDWs have been careless about not returning a zero rowcount estimate, apply clamp_row_est to whatever an FDW comes up with as the rows estimate. Per report from Andreas Seltenreich. Back-patch to 9.2. Earlier branches did not have the separation between set_base_rel_sizes and set_base_rel_pathlists steps, so there was no intermediate state where an appendrel would have had inconsistent rowcount and pathlist. It's possible that adding the Assert to set_rel_size would be a good idea in older branches too; but since they're not under development any more, it's likely not worth the trouble. http://git.postgresql.org/pg/commitdiff/358eaa01bf95935f9af968faf5b08d9914f6a445
  • Fix oversight in flattening of subqueries with empty FROM. I missed a restriction that commit f4abd0241de20d5d6a79b84992b9e88603d44134 should have enforced: we can't pull up an empty-FROM subquery if it's under an outer join, because then we'd need to wrap its output columns in PlaceHolderVars. As the code currently stands, the PHVs end up with empty relid sets, which doesn't work (and is correctly caught by an Assert). It's possible that this could be fixed by assigning the PHVs the relid sets of the parent FromExpr/JoinExpr, but getting that to work is more complication than I care to add right now; indeed it's likely that we'll never bother, since pulling up empty-FROM subqueries is a rather marginal optimization anyway. Per report from Andreas Seltenreich. Back-patch to 9.5 where the faulty code was added. http://git.postgresql.org/pg/commitdiff/fca8e59c1c582030dd7a3c870e1c3c70e8a193aa

Andres Freund pushed:

  • Fix bug around assignment expressions containing indirections. Handling of assigned-to expressions with indirection (e.g. set f1[1] = 3) was broken for ON CONFLICT DO UPDATE. The problem was that ParseState was consulted to determine if an INSERT-appropriate or UPDATE-appropriate behavior should be used when transforming expressions with indirections. When the wrong path was taken the old row was substituted with NULL, leading to wrong results.. To fix remove p_is_update and only use p_is_insert to decide how to transform the assignment expression, and uset p_is_insert while parsing the on conflict statement. This isn't particularly pretty, but it's not any worse than before. Author: Peter Geoghegan, slightly edited by me Discussion: CAM3SWZS8RPvA=KFxADZWw3wAHnnbxMxDzkEC6fNaFc7zSm411w@mail.gmail.com Backpatch: 9.5, where the feature was introduced http://git.postgresql.org/pg/commitdiff/c1ca3a19df376bcbb6d651d15b9a4ffcaa377ff1
  • Fix flattening of nested grouping sets. Previously nested grouping set specifications accidentally weren't flattened, but instead contained the nested specification as a element in the outer list. Fix this by, as actually documented in comments, concatenating the nested set specification into the outer one. Also add tests to prevent this from breaking again. Author: Andrew Gierth, with tests from Jeevan Chalke Reported-By: Jeevan Chalke Discussion: CAM2+6=V5YvuxB+EyN4iH=GbD-XTA435TCNvnDFSD--YvXs+pww@mail.gmail.com Backpatch: 9.5, where grouping sets were introduced http://git.postgresql.org/pg/commitdiff/faab14ecb8c1b4ea2bee3723d4fa04f47275abd3
  • Allow to push down clauses from HAVING to WHERE when grouping sets are used. Previously we disallowed pushing down quals to WHERE in the presence of grouping sets. That's overly restrictive. We now instead copy quals to WHERE if applicable, leaving the one in HAVING in place. That's because, at that stage of the planning process, it's nontrivial to determine if it's safe to remove the one in HAVING. Author: Andrew Gierth Discussion: 874mkt3l59.fsf@news-spur.riddles.org.uk Backpatch: 9.5, where grouping sets were introduced. This isn't exactly a bugfix, but it seems better to keep the branches in sync at this point. http://git.postgresql.org/pg/commitdiff/61444bfb809d3a088a270a59f383af3d4cd157b0
  • Build column mapping for grouping sets in all required cases. The previous coding frequently failed to fail because for one it's unusual to have rollup clauses with one column, and for another sometimes the wrong mapping didn't cause obvious problems. Author: Jeevan Chalke Reviewed-By: Andrew Gierth Discussion: CAM2+6=W=9=hQOipH0HAPbkun3Z3TFWij_EiHue0_6UX=oR=1kw@mail.gmail.com Backpatch: 9.5, where grouping sets were introduced http://git.postgresql.org/pg/commitdiff/144666f65b500fef864bca318f6245b03c0f457c
  • Recognize GROUPING() as a aggregate expression. Previously GROUPING() was not recognized as a aggregate expression, erroneously allowing the planner to move it from HAVING to WHERE. Author: Jeevan Chalke Reviewed-By: Andrew Gierth Discussion: CAM2+6=WG9omG5rFOMAYBweJxmpTaapvVp5pCeMrE6BfpCwr4Og@mail.gmail.com Backpatch: 9.5, where grouping sets were introduced http://git.postgresql.org/pg/commitdiff/e6d8cb77c029b8122607e3d2eb1f3fca36d7b1db
  • Check the relevant index element in ON CONFLICT unique index inference. ON CONFLICT unique index inference had a thinko that could affect cases where the user-supplied inference clause required that an attribute match a particular (user specified) collation and/or opclass. infer_collation_opclass_match() has to check for opclass and/or collation matches and that the attribute is in the list of attributes or expressions known to be in the definition of the index under consideration. The bug was that these two conditions weren't necessarily evaluated for the same index attribute. Author: Peter Geoghegan Discussion: CAM3SWZR4uug=WvmGk7UgsqHn2MkEzy9YU-+8jKGO4JPhesyeWg@mail.gmail.com Backpatch: 9.5, where ON CONFLICT was introduced http://git.postgresql.org/pg/commitdiff/159cff58cf3b565be3c17901698a74238e9e23f8

Joe Conway pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Peter Geoghegan sent in a patch to prefetch from memtuples array in tuplesort on compilers that support same.

Adam Brightwell sent in a patch to remove an unneeded #include in objectaddress.h.

Alexander Korotkov sent in two more revisions of a patch to allow pg_rewind to work when target timeline was switched.

Heikki Linnakangas sent in another revision of a patch to ensure that an all-zero GIN page doesn't cause an assertion failure.

Michael Paquier sent in two more revisions of a patch to fix the location of output logs of pg_regress.

Sameer Thakur sent in two more revisions of a patch to implement a VACUUM progress checker.

Petr Jelinek sent in three more revisions of a patch to allow CREATE EXTENSION to pull in any dependent extensions.

Haribabu Kommi sent in a patch to add a new function called pg_hba_lookup() to get all matching entries from the pg_hba.conf for the providing input data.

Peter Eisentraut sent in another revision of a patch to add --slot option to pg_basebackup.

Michael Paquier sent in a patch to fix a dump-restore hazard created by ALTER TABLE .. ADD PRIMARY KEY .. USING INDEX.

Jaimin Pan sent in a patch to make object_classes break loudly.

Peter Geoghegan sent in a patch to add a new tie-breaker which is used only when the tuples being sorted still fit in memory.

Paul Ramsey sent in three more revisions of a patch to allow specifying remote extensions in the PostgreSQL FDW.

Fabien COELHO sent in three more revisions of a patch to add pgbench stats per script.

Ildus Kurbangaliev sent in two more revisions of a patch to add a wait_event column to pg_stat_activity.

Dinesh Kumar sent in another revision of a patch to add an SQL-callable pg_report_log() function.

Fabrízio de Royes Mello and Michael Paquier traded patches to add CINE for ALTER TABLE ... ADD COLUMN.

Simon Riggs and Heikki Linnakangas traded patches to fix a WAL logging problem.

Etsuro Fujita sent in two revisions of a patch to support ForeignRecheck for late row locking.

Michael Paquier sent in a patch to fix some memory leaks in pg_rewind caused by missing PQclear calls.

Florent Guiliani added a replication command named LOGICAL_DECODING_SNAPSHOT that does everything CREATE_REPLICATION_SLOT does for logical slots except that the slot is automatically dropped.

Fabrízio de Royes Mello sent in a patch to add more test cases to cover ALTER TABLE on views.

Kyotaro HORIGUCHI sent in two more revisions of a patch to allow backslash-continuations in custom scripts for pgbench.

Ãlvaro Herrera sent in another revision of a patch to add valgrind tests to bufmgr.c.

Heikki Linnakangas and Michael Paquier traded patches to support TAP tests with MSVC and Windows.

Pavel Stěhule sent in another revision of a patch to add a --strict-names option to pg_dump.

Noah Misch sent in a patch to correctly raise an error in cases where IBM xlc is wrong.

Pavel Stěhule sent in two more revisions of a patch to add a libpq context filter.

Fabien COELHO sent in two more revisions of a patch to allow extending pgbench expressions with functions.

Fabien COELHO sent in another revision of a patch to add a checkpoint_sort GUC.

Heikki Linnakangas sent in another revision of a patch to allow sharing aggregate state.

Peter Geoghegan sent in a patch to remove a reference to an executor README section about speculative insertion.

par N Bougain le lundi 27 juillet 2015 à 20h08

mercredi 22 juillet 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 19 juillet 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juillet

PostgreSQL Local

  • Rubens Souza de 2ndQuadrant Italie organise un meetup "Comment installer PostgreSQL sur un Raspberry PI" le jeudi 23 juillet à Prato : http://goo.gl/YYpsy7
  • Le PGDay Campinas 2015 aura lieu à Campinas (Brésil) le 7 août : http://pgdaycampinas.com.br/english/
  • Le PostgresOpen 2015 aura lieu à Dallas (Texas, USA) du 16 au 18 septembre : http://2015.postgresopen.org/
  • La session PostgreSQL n°7 aura lieu le 24 septembre 2015 à Paris (France) : http://www.postgresql-sessions.org/7/about
  • Le PGDay.IT 2015 aura lieu à Prato le 23 octobre 2015. L'appel international à conférenciers court jusqu'au 8 août : http://pgday.it
  • PostgreSQL Conference Europe 2015 aura lieu du 27 au 30 octobre au Vienna Marriott Hotel à Vienne (Autriche). L'appel à conférenciers est lancé jusqu'au 7 août : http://2015.pgconf.eu/
  • PGConf Silicon Valley 2015 se tiendra au centre de convention sud de San Francisco les 17 & 18 novembre : http://www.pgconfsv.com
  • PGBR2015 (la PgConf brésilienne) aura lieu à Porto Alegre (État du Rio Grande do Sul) les 18, 19 et 20 novembre : http://pgbr.postgresql.org.br/2015/en/#call-for-papers
  • PostgreSQL Conference Europe 2015 aura lieu du 27 au 30 octobre au Vienna Marriott Hotel à Vienne (Autriche). L'appel à conférenciers est lancé jusqu'au 7 août : http://2015.pgconf.eu/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150719225824.GA4164@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Bruce Momjian pushed:

Heikki Linnakangas pushed:

  • Reformat code in ATPostAlterTypeParse. The code in ATPostAlterTypeParse was very deeply indented, mostly because there were two nested switch-case statements, which add a lot of indentation. Use if-else blocks instead, to make the code less indented and more readable. This is in preparation for next patch that makes some actualy changes to the function. These cosmetic parts have been separated to make it easier to see the real changes in the other patch. http://git.postgresql.org/pg/commitdiff/1ab9faaecb03e685aeeb16143c19c0a24d6b0048
  • Make regression test output stable. In the test query I added for ALTER TABLE retaining comments, the order of the result rows was not stable, and varied across systems. Add an ORDER BY to make the order predictable. This should fix the buildfarm failures. http://git.postgresql.org/pg/commitdiff/1a56498e5f6db949a066fb125199a7389a798421
  • Add ALTER OPERATOR command, for changing selectivity estimator functions. Other options cannot be changed, as it's not totally clear if cached plans would need to be invalidated if one of the other options change. Selectivity estimator functions only change plan costs, not correctness of plans, so those should be safe. Original patch by Uriy Zhuravlev, heavily edited by me. http://git.postgresql.org/pg/commitdiff/321eed5f0f7563a0cabb3d7a98132856287c1ad1
  • Fix event trigger support for the new ALTER OPERATOR command. Also, the lock on pg_operator should not be released until end of transaction. http://git.postgresql.org/pg/commitdiff/d5c0495cd4b9c78fdfc00961f4ae14c39f877f59
  • Retain comments on indexes and constraints at ALTER TABLE ... TYPE ... When a column's datatype is changed, ATExecAlterColumnType() rebuilds all the affected indexes and constraints, and the comments from the old indexes/constraints were not carried over. To fix, create a synthetic COMMENT ON command in the work queue, to re-add any comments on constraints. For indexes, there's a comment field in IndexStmt that is used. This fixes bug #13126, reported by Kirill Simonov. Original patch by Michael Paquier, reviewed by Petr Jelinek and me. This bug is present in all versions, but only backpatch to 9.5. Given how minor the issue is, it doesn't seem worth the work and risk to backpatch further than that. http://git.postgresql.org/pg/commitdiff/e42375fc8124e99c33fa330c53c2b4b502fa0baf

Fujii Masao pushed:

Álvaro Herrera pushed:

Robert Haas pushed:

Noah Misch pushed:

  • AIX: Link the postgres executable with -Wl,-brtllib. This allows PostgreSQL modules and their dependencies to have undefined symbols, resolved at runtime. Perl module shared objects rely on that in Perl 5.8.0 and later. This fixes the crash when PL/PerlU loads such modules, as the hstore_plperl test suite does. Module authors can link using -Wl,-G to permit undefined symbols; by default, linking will fail as it has. Back-patch to 9.0 (all supported versions). http://git.postgresql.org/pg/commitdiff/bcd7c41206faf6d9654aa6e3766f87770d4fb305
  • MinGW: Link ltree_plpython with plpython. The MSVC build system already did this, and building against Python 3 requires it. Back-patch to 9.5, where the module was introduced. http://git.postgresql.org/pg/commitdiff/736c1f238b3eeaf0f1cecf1753eb5194367fbad9
  • AIX: Link TRANSFORM modules with their dependencies. The result closely resembles linking of these modules for the "win32" port. Augment the $(exports_file) header so the file is also usable as an import file. Unfortunately, relocating an AIX installation will now require adding $(pkglibdir) to LD_LIBRARY_PATH. Back-patch to 9.5, where the modules were introduced. http://git.postgresql.org/pg/commitdiff/7193436744819270eeb772f6ada4ec7a388c0b5f
  • AIX: Test the -qlonglong option before use. xlc provides "long long" unconditionally at C99-compatible language levels, and this option provokes a warning. The warning interferes with "configure" tests that fail in response to any warning. Notably, before commit 85a2a8903f7e9151793308d0638621003aded5ae, it interfered with the test for -qnoansialias. Back-patch to 9.0 (all supported versions). http://git.postgresql.org/pg/commitdiff/43d89a23d59c487bc9258fad7a6187864cb8c0c0

Magnus Hagander pushed:

Tom Lane pushed:

  • Fix a low-probability crash in our qsort implementation. It's standard for quicksort implementations, after having partitioned the input into two subgroups, to recurse to process the smaller partition and then handle the larger partition by iterating. This method guarantees that no more than log2(N) levels of recursion can be needed. However, Bentley and McIlroy argued that checking to see which partition is smaller isn't worth the cycles, and so their code doesn't do that but just always recurses on the left partition. In most cases that's fine; but with worst-case input we might need O(N) levels of recursion, and that means that qsort could be driven to stack overflow. Such an overflow seems to be the only explanation for today's report from Yiqing Jin of a SIGSEGV in med3_tuple while creating an index of a couple billion entries with a very large maintenance_work_mem setting. Therefore, let's spend the few additional cycles and lines of code needed to choose the smaller partition for recursion. Also, fix up the qsort code so that it properly uses size_t not int for some intermediate values representing numbers of items. This would only be a live risk when sorting more than INT_MAX bytes (in qsort/qsort_arg) or tuples (in qsort_tuple), which I believe would never happen with any caller in the current core code --- but perhaps it could happen with call sites in third-party modules? In any case, this is trouble waiting to happen, and the corrected code is probably if anything shorter and faster than before, since it removes sign-extension steps that had to happen when converting between int and size_t. In passing, move a couple of CHECK_FOR_INTERRUPTS() calls so that it's not necessary to preserve the value of "r" across them, and prettify the output of gen_qsort_tuple.pl a little. Back-patch to all supported branches. The odds of hitting this issue are probably higher in 9.4 and up than before, due to the new ability to allocate sort workspaces exceeding 1GB, but there's no good reason to believe that it's impossible to crash older branches this way. http://git.postgresql.org/pg/commitdiff/9d6077abf9d6efd992a59f05ef5aba981ea32096
  • Fix entirely broken permissions test in new alter_operator regression test. Not only did this test fail to test what it was supposed to test, but it left a user definition lying around, which caused subsequent runs of the regression tests to fail. http://git.postgresql.org/pg/commitdiff/266e771435bfed648138f6b684c895c8225dc8fc
  • Repair mishandling of cached cast-expression trees in plpgsql. In commit 1345cc67bbb014209714af32b5681b1e11eaf964, I introduced caching of expressions representing type-cast operations into plpgsql. However, I supposed that I could cache both the expression trees and the evaluation state trees derived from them for the life of the session. This doesn't work, because we execute the expressions in plpgsql's simple_eval_estate, which has an ecxt_per_query_memory that is only transaction-lifespan. Therefore we can end up putting pointers into the evaluation state tree that point to transaction-lifespan memory; in particular this happens if the cast expression calls a SQL-language function, as reported by Geoff Winkless. The minimum-risk fix seems to be to treat the state trees the same way we do for "simple expression" trees in plpgsql, ie create them in the simple_eval_estate's ecxt_per_query_memory, which means recreating them once per transaction. Since I had to introduce bookkeeping overhead for that anyway, I bought back some of the added cost by sharing the read-only expression trees across all functions in the session, instead of using a per-function table as originally. The simple-expression bookkeeping takes care of the recursive-usage risk that I was concerned about avoiding before. At some point we should take a harder look at how all this works, and see if we can't reduce the amount of tree reinitialization needed. But that won't happen for 9.5. http://git.postgresql.org/pg/commitdiff/0fc94a5bab4d0155db5d15197ed3bd8cb435eb21
  • Make WaitLatchOrSocket's timeout detection more robust. In the previous coding, timeout would be noticed and reported only when poll() or socket() returned zero (or the equivalent behavior on Windows). Ordinarily that should work well enough, but it seems conceivable that we could get into a state where poll() always returns a nonzero value --- for example, if it is noticing a condition on one of the file descriptors that we do not think is reason to exit the loop. If that happened, we'd be in a busy-wait loop that would fail to terminate even when the timeout expires. We can make this more robust at essentially no cost, by deciding to exit of our own accord if we compute a zero or negative time-remaining-to-wait. Previously the code noted this but just clamped the time-remaining to zero, expecting that we'd detect timeout on the next loop iteration. Back-patch to 9.2. While 9.1 had a version of WaitLatchOrSocket, it was primitive compared to later versions, and did not guarantee reliable detection of timeouts anyway. (Essentially, this is a refinement of commit 3e7fdcffd6f77187, which was back-patched only as far as 9.2.) http://git.postgresql.org/pg/commitdiff/576a95b3a1ce465066c38d6859ccf64fca656e49

Andrew Dunstan pushed:

  • Support JSON negative array subscripts everywhere Previously, there was an inconsistency across json/jsonb operators that operate on datums containing JSON arrays -- only some operators supported negative array count-from-the-end subscripting. Specifically, only a new-to-9.5 jsonb deletion operator had support (the new "jsonb - integer" operator). This inconsistency seemed likely to be counter-intuitive to users. To fix, allow all places where the user can supply an integer subscript to accept a negative subscript value, including path-orientated operators and functions, as well as other extraction operators. This will need to be called out as an incompatibility in the 9.5 release notes, since it's possible that users are relying on certain established extraction operators changed here yielding NULL in the event of a negative subscript. For the json type, this requires adding a way of cheaply getting the total JSON array element count ahead of time when parsing arrays with a negative subscript involved, necessitating an ad-hoc lex and parse. This is followed by a "conversion" from a negative subscript to its equivalent positive-wise value using the count. From there on, it's as if a positive-wise value was originally provided. Note that there is still a minor inconsistency here across jsonb deletion operators. Unlike the aforementioned new "-" deletion operator that accepts an integer on its right hand side, the new "#-" path orientated deletion variant does not throw an error when it appears like an array subscript (input that could be recognized by as an integer literal) is being used on an object, which is wrong-headed. The reason for not being stricter is that it could be the case that an object pair happens to have a key value that looks like an integer; in general, these two possibilities are impossible to differentiate with rhs path text[] argument elements. However, we still don't allow the "#-" path-orientated deletion operator to perform array-style subscripting. Rather, we just return the original left operand value in the event of a negative subscript (which seems analogous to how the established "jsonb/json #> text[]" path-orientated operator may yield NULL in the event of an invalid subscript). In passing, make SetArrayPath() stricter about not accepting cases where there is trailing non-numeric garbage bytes rather than a clean NUL byte. This means, for example, that strings like "10e10" are now not accepted as an array subscript of 10 by some new-to-9.5 path-orientated jsonb operators (e.g. the new #- operator). Finally, remove dead code for jsonb subscript deletion; arguably, this should have been done in commit b81c7b409. Peter Geoghegan and Andrew Dunstan http://git.postgresql.org/pg/commitdiff/e02d44b8a74810341c90add4cd49e428b9d406b9
  • Release note compatibility item. Note that json and jsonb extraction operators no longer consider a negative subscript to be invalid. http://git.postgresql.org/pg/commitdiff/473865048517c7808ddcf2299d054d8fe30fc6d5
  • Enable transforms modules to build and test on Cygwin. This still doesn't work correctly with Python 3, but I am committing this so we can get Cygwin buildfarm members building with Python 2. http://git.postgresql.org/pg/commitdiff/00eff86cb8c2c9de9197197b4176362d1433f8f6
  • Remove dead code. Defect noticed by Coverity. http://git.postgresql.org/pg/commitdiff/9aa663463bbf123e9d38dab88eeaef981fbc6caf

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Haribabu Kommi sent in two more revisions of a patch to help improve the performance of vacuum truncation scans.

Dinesh Kumar sent in a patch to add an SQL function to report a log message.

Jeevan Chalke sent in a patch to fix some infelicities in GROUPING SETS.

Jeff Janes sent in another revision of a patch to make pg_trgm work better with GIN.

Amit Kapila sent in another revision of a patch to add machinery for assessing parallel safety.

Paul Ramsey sent in two revisions of a patch to allow specifying extensions understood to be installed in PostgreSQL foreign servers.

Jeevan Chalke, Kyotaro HORIGUCHI, and Andrew Gierth traded patches to fix an issue with GROUPING SETS that resulted in the not especially helpful, "unrecognized node type" error.

Sameer Thakur and Rahila Syed traded patches to provide a VACUUM progress checker.

Heikki Linnakangas sent in a patch to fix LWLock "variable" support broken by the lwlock scalability patch.

SAWADA Masahiko sent in another revision of a patch to avoid freezing very large tables without need.

Kyotaro HORIGUCHI sent in another revision of a patch to implement multivariate statistics.

Michael Paquier sent in a patch to ensure that pg_rewind ignores xlog.

Peter Geoghegan sent in two revisions of a patch to use software-based memory prefetching while sequentially fetching from SortTuple array and tuplestore.

Jeevan Chalke sent in a patch to fix a collation bug in GROUPING SETS.

Brendan Jurd sent in another revision of a patch to add pg_notification_queue_usage().

Pavel Stehule sent in a patch to add tab completion to DROP POLICY in psql.

Fabien COELHO sent in two revisions of a patch to add per-script statistics and other improvements to pgbench.

Michael Paquier sent in another revision of a patch to retain comments on indexes and constraints after issuing ALTER TABLE.

Petr Jelinek sent in another revision of a patch to allow adding an extension including its dependencies.

Julien Rouhaud sent in a patch to allow setting effective_io_concurrency per tablespace.

Álvaro Herrera sent in a patch to ensure that breakage of object_class be obvious.

Andres Freund sent in two revisions of a patch to make heap extension saner.

Pavel Stehule sent in another revision of a patch to add a --strict option to pg_dump.

par N Bougain le mercredi 22 juillet 2015 à 07h21

mardi 14 juillet 2015

Guillaume Lelarge

Comment quantifier le maintenance_work_mem

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

Je suis en train d'écrire le chapitre sur la maintenance. Parmi les opérations de maintenance se trouve l'ordre VACUUM. Beaucoup de choses ont déjà été écrites dans le livre sur le VACUUM mais j'avais bizarrement oublié une chose. Une bonne configuration du paramètre maintenance_work_mem permet d'avoir un VACUUM performant. Mais comment peut-on savoir que la valeur du maintenance_work_mem est suffisante ?

J'ai donc creusé hier soir dans les sources de PostgreSQL à la recherche de ce qui est stocké dans cette mémoire. Tout se trouve dans src/backend/commands/vacuumlazy.c, principalement dans la fonction lazy_space_alloc(). En gros, PostgreSQL y place un tableau de la structure ItemPointerData. Cette structure prend six octets. Donc une estimation (très grosse) serait de dire qu'on peut stocker maintenance_work_mem/6 positions d'enregistrements morts dans cette mémoire. Un patch rapide (voir le fichier joint) nous prouve cette théorie :

Nous plaçons le paramètre client_min_messages au niveau log pour voir les traces ajoutées par le patch :

postgres=# SET client_min_messages TO log;
SET

Nous créons la table et désactivons l'autovacuum sur cette table pour le gérer nous-même :

postgres=# DROP TABLE IF EXISTS t1;
DROP TABLE
postgres=# CREATE TABLE t1(id INTEGER PRIMARY KEY);
CREATE TABLE
postgres=# ALTER TABLE t1 SET (autovacuum_enabled = OFF);
ALTER TABLE

Nous insérons un million de lignes, puis en supprimons 900000 :

postgres=# INSERT INTO t1 SELECT generate_series(1, 1000000);
INSERT 0 1000000
postgres=# DELETE FROM t1 WHERE id<900000;
DELETE 899999

Nous configurons maintenance_work_mem à 1 Mo (en fait, suffisamment petit pour voir que le VACUUM a besoin de plusieurs passes dû au manque de mémoire) :

postgres=# SET maintenance_work_mem TO '1MB';
SET
postgres=# VACUUM t1;
LOG:  patch - vac_work_mem: 1024
LOG:  patch - sizeof(ItemPointerData): 6
LOG:  patch - maxtuples: 174762
LOG:  patch - step 1
LOG:  patch - step 2
LOG:  patch - step 3
LOG:  patch - step 4
LOG:  patch - step 5
LOG:  patch - step 6
VACUUM

La fonction de calcul de la taille mémoire a bien noté le maintenance_work_mem à 1 Mo (1024 Ko). La taille de la structure est bien de 6 octets. Il est donc possible de stocker 1024*1024/6 enregistrements, soit 174762 enregistrements. Ayant supprimé 900000 enregistrements, il me faut 6 passes (l'arrondi supérieur de l'opération 900000/174762) pour traiter la table entière. Pas efficace.

Essayons dans les mêmes conditions mais avec un maintenance_work_mem trois fois plus gros :

postgres=# TRUNCATE t1;
TRUNCATE TABLE
postgres=# INSERT INTO t1 SELECT generate_series(1, 1000000);
INSERT 0 1000000
postgres=# DELETE FROM t1 WHERE id<900000;
DELETE 899999
postgres=# SET maintenance_work_mem TO '3MB';
SET
postgres=# VACUUM t1;
LOG:  patch - vac_work_mem: 3072
LOG:  patch - sizeof(ItemPointerData): 6
LOG:  patch - maxtuples: 524288
LOG:  patch - step 1
LOG:  patch - step 2
VACUUM

Nous ne faisons plus que deux passes (tout d'abord 524288 enregistrements, puis 375711), c'est plus efficace mais non optimal.

Essayons maintenant avec le maintenance_work_mem de base (64 Mo) :

postgres=# TRUNCATE t1;
TRUNCATE TABLE
postgres=# INSERT INTO t1 SELECT generate_series(1, 1000000);
INSERT 0 1000000
postgres=# DELETE FROM t1 WHERE id<900000;
DELETE 899999
postgres=# RESET maintenance_work_mem;
RESET
postgres=# VACUUM VERBOSE t1;
INFO:  vacuuming "public.t1"
LOG:  patch - vac_work_mem: 65536
LOG:  patch - sizeof(ItemPointerData): 6
LOG:  patch - maxtuples: 1287675
LOG:  patch - step 1
VACUUM

Seule une passe est réalisée. Il est à noter que la mémoire prise ne correspond pas au 64 Mo. 64 Mo me permet de stocker 11 millions d'enregistrements morts, mais je n'ai dans la table que 1000000 d'enregistrements dont 90% est mort. Autrement dit, j'ai besoin de beaucoup moins de mémoire. C'est bien le cas ici où, au lieu de 11 millions d'enregistrements, on peut en stocker 1287675 (soit un peu plus de 7 Mo).

De tout ça, comment puis-je savoir si mon maintenance_work_mem est bien configuré ? Il faut se baser sur le nombre d'enregistrements (morts) contenus dans les tables. Ça correspond à cette requête pour les tables de ma base de connexion :

SELECT pg_size_pretty(max(n_dead_tup*6)) AS custom_maintenance_work_mem
FROM pg_stat_all_tables;

Dans l'exemple précédent, cela me donnerait ceci :

postgres=# TRUNCATE t1;
TRUNCATE TABLE
postgres=# INSERT INTO t1 SELECT generate_series(1, 1000000);
INSERT 0 1000000
postgres=# DELETE FROM t1 WHERE id<900000;
DELETE 899999
postgres=# SELECT pg_size_pretty(max(n_dead_tup*6)) AS custom_maintenance_work_mem,
           current_setting('maintenance_work_mem') AS current_maintenance_work_mem
          FROM pg_stat_all_tables;

 custom_maintenance_work_mem | current_maintenance_work_mem 
-----------------------------+------------------------------
 5273 kB                     | 64MB
(1 row)

Il me faut au minimum 5,2 Mo. Je suis donc tranquille.

Évidemment, le nombre d'enregistrements morts évolue dans le temps et il est tout à fait possible que la quantité de mémoire nécessaire soit bien plus importante. On peut se baser sur le nombre d'enregistrements total pour avoir le pire des cas comme ici :

b1=# SELECT pg_size_pretty(max((n_live_tup+n_dead_tup)*6)) AS custom_maintenance_work_mem,
     current_setting('maintenance_work_mem') AS current_maintenance_work_mem
     FROM pg_stat_all_tables;

 custom_maintenance_work_mem | current_maintenance_work_mem 
-----------------------------+------------------------------
 472 MB                      | 512MB
(1 row)

Ce qui révèle donc une configuration adéquate pour cet utilisateur.

par Guillaume Lelarge le mardi 14 juillet 2015 à 07h54

lundi 13 juillet 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 12 juillet 2015

Le PGDay.IT 2015 aura lieu à Prato le 23 octobre 2015. L'appel international à conférenciers court jusqu'au 8 août : http://pgday.it

La 2ème conférence PostgreSQL russe officielle se tiendra à Saint-Pétersbourg. Un meetup PostgreSQLRussia pour les nouveaux utilisateurs est programmé pour le 17 juillet, même endroit : http://PgDay.ru http://PostgreSQLRussia.org

Rubens Souza de 2ndQuadrant Italie organise un meetup "Comment installer PostgreSQL sur un Raspberry PI" le jeudi 23 juillet à Prato : http://goo.gl/YYpsy7

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juillet

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150712232259.GB2535@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Andres Freund pushed:

  • Fix logical decoding bug leading to inefficient reopening of files. When spilling transaction data to disk a simple typo caused the output file to be closed and reopened for every serialized change. That happens to not have a huge impact on linux, which is why it probably wasn't noticed so far, but on windows that appears to trigger actual disk writes after every change. Not fun. The bug fortunately does not have any impact besides speed. A change could end up being in the wrong segment (last instead of next), but since we read all files to the end, that's just ugly, not really problematic. It's not a problem to upgrade, since transaction spill files do not persist across restarts. Bug: #13484 Reported-By: Olivier Gosseaume Discussion: 20150703090217.1190.63940@wrigleys.postgresql.org Backpatch to 9.4, where logical decoding was added. http://git.postgresql.org/pg/commitdiff/b2f6f749c7a5936adbb555e248e8e4df35c00a4a
  • Fix pg_recvlogical not to fsync output when it's a tty or pipe. The previous coding tried to handle possible failures when fsyncing a tty or pipe fd by accepting EINVAL - but apparently some platforms (windows, OSX) don't reliably return that. So instead check whether the output fd refers to a pipe or a tty when opening it. Reported-By: Olivier Gosseaume, Marko Tiikkaja Discussion: 559AF98B.3050901@joh.to Backpatch to 9.4, where pg_recvlogical was added. http://git.postgresql.org/pg/commitdiff/5c0de384d2ceceb07e77e1368e07868244be6762
  • Add psql PROMPT variable showing the pid of the connected to backend. The substitution for the pid is %p. Author: Julien Rouhaud Discussion: 116262CF971C844FB6E793F8809B51C6E99D48@BPXM02GP.gisp.nec.co.jp http://git.postgresql.org/pg/commitdiff/275f05c990c46f8dfe3cb46a3279521bda9e9e27
  • Refer to %p in the psql docs as 'process ID' not 'pid'. Per Tom Lane. http://git.postgresql.org/pg/commitdiff/4af04f96bc9f604a57bf829469b26d8513fd6401
  • Optionally don't error out due to preexisting slots in commandline utilities. pg_receivexlog and pg_recvlogical error out when --create-slot is specified and a slot with the same name already exists. In some cases, especially with pg_receivexlog, that's rather annoying and requires additional scripting. Backpatch to 9.5 as slot control functions have newly been added to pg_receivexlog, and there doesn't seem much point leaving it in a less useful state. Discussion: 20150619144755.GG29350@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/ff27db5dd2fc096d89d3f995d3f650ec6d3bc147
  • For consistency add a pfree to ON CONFLICT set_plan_refs code. Backpatch to 9.5 where ON CONFLICT was introduced. Author: Peter Geoghegan http://git.postgresql.org/pg/commitdiff/3ed26e5f87a90bedaa3d7feb9e197e0d9f3fb252

Fujii Masao pushed:

Heikki Linnakangas pushed:

  • Improve handling of out-of-memory in libpq. If an allocation fails in the main message handling loop, pqParseInput3 or pqParseInput2, it should not be treated as "not enough data available yet". Otherwise libpq will wait indefinitely for more data to arrive from the server, and gets stuck forever. This isn't a complete fix - getParamDescriptions and getCopyStart still have the same issue, but it's a step in the right direction. Michael Paquier and me. Backpatch to all supported versions. http://git.postgresql.org/pg/commitdiff/414bef30bfab20451e15fe799642b52166db8c34
  • Move pthread-tests earlier in the autoconf script. On some Linux systems, "-lrt" exposed pthread-functions, so that linking with -lrt was seemingly enough to make a program that uses pthreads to work. However, when linking libpq, the dependency to libpthread was not marked correctly, so that when an executable was linked with -lpq but without -pthread, you got errors about undefined pthread_* functions from libpq. To fix, test for the flags required to use pthreads earlier in the autoconf script, before checking any other libraries. This should fix the failure on buildfarm member shearwater. gharial is also failing; hopefully this fixes that too although the failure looks somewhat different. http://git.postgresql.org/pg/commitdiff/3b14a17c8e60e8ae9227b9533667743508056c35
  • Use AS_IF rather than plain shell "if" in pthread-check. Autoconf generates additional code for the first AC_CHECK_HEADERS call in the script. If the first call is within an if-block, the additional code is put inside the if-block too, even though it is needed by subsequent AC_CHECK_HEADERS checks and should always be executed. When I moved the pthread-related checks earlier in the script, the pthread.h test inside the block became the very first AC_CHECK_HEADERS call in the script, triggering that problem. To fix, use AS_IF instead of plain shell if. AS_IF knows about that issue, and makes sure the additional code is always executed. To be completely safe from this issue (and others), we should always be using AS_IF instead of plain if, but that seems like excessive caution given that this is the first time we have trouble like this. Plain if-then is more readable than AS_IF. This should fix compilation with --disable-thread-safety, and hopefully the buildfarm failure on forgmouth, related to mingw standard headers, too. I backpatched the previous fixes to 9.5, but it's starting to look like these changes are too fiddly to backpatch, so commit this to master only, and revert all the pthread-related configure changes in 9.5. http://git.postgresql.org/pg/commitdiff/01051a9879fcd353eaf0d3788a911e774b52798c
  • Improve logging of TAP tests. Create a log file for each test run. Stdout and stderr of the test script, as well as any subprocesses run as part of the test, are redirected to the log file. This makes it a lot easier to debug test failures. Also print the test output (ok 12 - ... messages) to the log file, and the command line of any external programs executed with the system_or_bail and run_log functions. This makes it a lot easier to debug failing tests. Modify some of the pg_ctl and other command invocations to not use 'silent' or 'quiet' options, and don't redirect output to /dev/null, so that you get all the information in the log instead. In the passing, construct some command lines in a way that works if $tempdir contains quote-characters. I haven't systematically gone through all of them or tested that, so I don't know if this is enough to make that work. pg_rewind tests had a custom mechanism for creating a similar log file. Use the new generic facility instead. Michael Paquier and me. http://git.postgresql.org/pg/commitdiff/1ea06203b82b98b5098808667f6ba652181ef5b2
  • Fix another broken link in documentation. Tom fixed another one of these in commit 7f32dbcd, but there was another almost identical one in libpq docs. Per his comment: HP's web server has apparently become case-sensitive sometime recently. Per bug #13479 from Daniel Abraham. Corrected link identified by Alvaro. http://git.postgresql.org/pg/commitdiff/aaec6a6d37b664199fd7744b976a7dc912ae000a
  • Use --debug flag in "remote" pg_rewind regression tests. Gives more information in the log, to debug possible failures. http://git.postgresql.org/pg/commitdiff/23b8928829038ef3fba5a04e4f2707c6034464c4
  • Replace our hacked version of ax_pthread.m4 with latest upstream version. Our version was different from the upstream version in that we tried to use all possible pthread-related flags that the compiler accepts, rather than just the first one that works. That change was made in commit e48322a6d6cfce1ec52ab303441df329ddbc04d1, to work-around a bug affecting GCC versions 3.2 and below (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=8888), although we didn't realize that it was a GCC bug at the time. We hardly care about that old GCC versions anymore, so we no longer need that workaround. This fixes the macro for compilers that print warnings with the chosen flags. That's pretty annoying on its own right, but it also inconspicuously disabled thread-safety, because we refused to use any pthread-related flags if the compiler produced warnings. Max Filippov reported that problem when linking with uClibc and OpenSSL. The warnings-check was added because the workaround for the GCC bug caused warnings otherwise, so it's no longer needed either. We can just use the upstream version as is. If you really want to compile with GCC version 3.2 or older, you can still work-around it manually by setting PTHREAD_CFLAGS="-pthread -lpthread" manually on the configure command line. Backpatch to 9.5. I don't want to unnecessarily rock the boat on stable branches, but 9.5 seems like fair game. http://git.postgresql.org/pg/commitdiff/e97af6c8bfc80d084bca5bf41f036de944b63efe
  • Copy-edit the docs changes of OWNER TO CURRENT/SESSION_USER additions. Commit 31eae602 added new syntax to many DDL commands to use CURRENT_USER or SESSION_USER instead of role name in ALTER ... OWNER TO, but because of a misplaced '{', the syntax in the docs implied that the syntax was "ALTER ... CURRENT_USER", instead of "ALTER ... OWNER TO CURRENT_USER". Fix that, and also the funny indentation in some of the modified syntax blurps. http://git.postgresql.org/pg/commitdiff/cba045b0bd25285242936fd678bc443bfd0d5b83

Tom Lane pushed:

  • Fix portability issue in pg_upgrade test script: avoid $PWD. SUSv2-era shells don't set the PWD variable, though anything more modern does. In the buildfarm environment this could lead to test.sh executing with PWD pointing to $HOME or another high-level directory, so that there were conflicts between concurrent executions of the test in different branch subdirectories. This appears to be the explanation for recent intermittent failures on buildfarm members binturong and dingo (and might well have something to do with the buildfarm script's failure to capture log files from pg_upgrade tests, too). To fix, just use `pwd` in place of $PWD. AFAICS test.sh is the only place in our source tree that depended on $PWD. Back-patch to all versions containing this script. Per buildfarm. Thanks to Oskari Saarenmaa for diagnosing the problem. http://git.postgresql.org/pg/commitdiff/9a8f58331067e18a5dc10670e687f21ae6a2172e
  • Fix postmaster's handling of a startup-process crash. Ordinarily, a failure (unexpected exit status) of the startup subprocess should be considered fatal, so the postmaster should just close up shop and quit. However, if we sent the startup process a SIGQUIT or SIGKILL signal, the failure is hardly "unexpected", and we should attempt restart; this is necessary for recovery from ordinary backend crashes in hot-standby scenarios. I attempted to implement the latter rule with a two-line patch in commit 442231d7f71764b8c628044e7ce2225f9aa43b67, but it now emerges that that patch was a few bricks shy of a load: it failed to distinguish the case of a signaled startup process from the case where the new startup process crashes before reaching database consistency. That resulted in infinitely respawning a new startup process only to have it crash again. To handle this properly, we really must track whether we have sent the *current* startup process a kill signal. Rather than add yet another ad-hoc boolean to the postmaster's state, I chose to unify this with the existing RecoveryError flag into an enum tracking the startup process's state. That seems more consistent with the postmaster's general state machine design. Back-patch to 9.0, like the previous patch. http://git.postgresql.org/pg/commitdiff/45811be94e8539190b5e1a4f2cbdfef97fa391b5
  • Improve documentation about array concat operator vs. underlying functions. The documentation implied that there was seldom any reason to use the array_append, array_prepend, and array_cat functions directly. But that's not really true, because they can help make it clear which case is meant, which the || operator can't do since it's overloaded to represent all three cases. Add some discussion and examples illustrating the potentially confusing behavior that can ensue if the parser misinterprets what was meant. Per a complaint from Michael Herold. Back-patch to 9.2, which is where || started to behave this way. http://git.postgresql.org/pg/commitdiff/e4f29ce32391525629c75aade86f2f939956c69c
  • Add now-required #include. Fixes compiler warning induced by 808ea8fc7bb259ddd810353719cac66e85a608c8. http://git.postgresql.org/pg/commitdiff/0a0fe2ff6ef65e3a1cf4d83d96eab144477a0220
  • Fix assorted memory leaks. Per Coverity (not that any of these are so non-obvious that they should not have been caught before commit). The extent of leakage is probably minor to unnoticeable, but a leak is a leak. Back-patch as necessary. Michael Paquier http://git.postgresql.org/pg/commitdiff/bcc87b6b00de5b36984e0b43a78a8377a3577548

Joe Conway pushed:

  • Improve regression test coverage of table lock modes vs permissions. Test the interactions with permissions and LOCK TABLE. Specifically ROW EXCLUSIVE, ACCESS SHARE, and ACCESS EXCLUSIVE modes against SELECT, INSERT, UPDATE, DELETE, and TRUNCATE permissions. Discussed by Stephen Frost and Michael Paquier, patch by the latter. Backpatch to 9.5 where matching behavior was first committed. http://git.postgresql.org/pg/commitdiff/e66a45344ff33d64aa6ff50673ff9fe8577ea6db
  • Add assign_expr_collations() to CreatePolicy() and AlterPolicy(). As noted by Noah Misch, CreatePolicy() and AlterPolicy() omit to call assign_expr_collations() on the node trees. Fix the omission and add his test case to the rowsecurity regression test. http://git.postgresql.org/pg/commitdiff/808ea8fc7bb259ddd810353719cac66e85a608c8

Noah Misch pushed:

Bruce Momjian pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Pavel Stehule sent in another revision of a patch to allow raw output from COPY in libpq.

Ashutosh Bapat sent in a patch to set up a notification channels for background workers, which can among other things spawn other background workers.

Julien Rouhaud sent in another revision of a patch to add an auto_explain sample ratio parameter.

Gurjeet Singh sent in a patch to add SlotIsPhyscial/SlotIsLogical macros and add a boolean for whether immediately to reserve an LSN for CREATE_REPLICATION_SLOT ... PHYSICAL.

David Rowley sent in another revision of a patch to improve performance for joins where outer side is unique.

Michael Paquier sent in another revision of a patch to add a TAP test for pg_dump checking data dump of extension tables.

Pavel Stehule and Merlin Moncure traded patches to add a client_min_messages_context GUC.

SAWADA Masahiko sent in four more revisions of a patch to a "frozen" bit to the visibility map.

Marko Tiikkaja sent in two revisions of a patch to expose confirmed_flush in pg_stat_replication_slots.

Pavel Stehule sent in a patch to allow tab completing boolean and enum variables if no specific tab completion already exists.

Amit Langote sent in a patch to correct some comments in predtest.c.

Satoshi Nagayasu sent in a patch to expose the log_disconnections GUC variable outside postgres.c.

Ildus Kurbangaliev sent in a patch to help monitor waits.

David Rowley sent in another revision of a patch to allow an aggregate's state to be shared among other aggregate functions when both aggregate's transition functions (and a few other things) match.

Ashutosh Bapat sent in a patch to create a transaction manager for FDWs.

Michael Paquier sent in another revision of a patch to support TAP tests with MSVC and Windows.

Stephen Frost sent in a patch to help secure copy.c handling for RLS.

Alexander Korotkov and Michael Paquier traded patches to add a fillfactor for GIN indexes.

Kyotaro HORIGUCHI sent in a flock of patches implementing asynchronous execution on FDWs.

Etsuro Fujita sent in a patch to modify create_foreignscan_plan so that it detects whether any system columns are requested if scanning a base relation.

Heikki Linnakangas sent in another revision of a patch to add some functionality to ALTER OPERATOR.

Peter Geoghegan sent in a patch to reuse abbreviated keys during second pass of ordered [set] aggregates.

Jeff Davis sent in another revision of a patch to implement memory accounting for hash aggregates.

Tomas Vondra sent in a patch to enable index-only scans with partial indexes.

Joe Conway sent in a patch to fix some infelicities in the interaction between RLS and collations.

Pavel Stehule and Alexander Shulgin traded patches to add generalized JSON output functions.

Michael Paquier sent in two more revisions of a patch to remove SSL renegotiation.

Michael Paquier sent in a patch to fix some memory leaks.

par N Bougain le lundi 13 juillet 2015 à 10h24

jeudi 9 juillet 2015

Damien Clochard

PostgreSQL 9.5 Alpha 1

Dans la chaleur de la semaine dernière, la première mouture de PostgreSQL 9.5 est sortie en toute discrétion. Pourtant cette première version alpha apporte une série de nouveautés impressionnantes !

Voici un petit tour d’horizon de ce qui vous attend…

Il est impossible de dresser une liste exhaustive des nouveautés et la sélection ci-dessous est tout à fait subjective :

  • La commande IMPORT FOREIGN SCHEMA va complètement changer la donne en ssimplifiant l’intégration de données (voir mon article sur les 70 nuances Postgres

  • Les Index Block-Range (BRIN) proposent une nouvelle méthode d’indexation ultra-compacte très utile pour les gros volumes de données

  • La sécurité au niveaux des lignes ( ‘‘Row Level Security’’ ) permet désromais de définir des politiques d’accès en fonction du contenu d’une table.

  • La commande INSERT ON CONFLICT UPDATE plus connue sous le nom ‘‘UPSERT’’ était une fonctionnalité très attendue

  • Il est maintenant possible d’utiliser l’héritage avec les tables étrangères. En clair il est désormais possible d’exporter une table partitionnée sur une instance distante et ainsi externaliser la croissance de vos bases (“scale-out”)

  • Une nuée de nouvelles fonctions JSONB

Pour une liste plus détaillée, rendez-vous sur la page What’s new in PostgreSQL 9.5

A vous de jouer !

Si vous aimez PostgreSQL mais que vous ne savez pas comment contriber, voici une belle occasion !

Les développeurs de PostgreSQL ont suspendu l’intégration de nouvelles fonctionnalités dans PostgreSQL et ils vont consacrer les prochaine semaines à la stabilisation du code.

Pour les aider, il suffit d’installer, de compiler et de tester PostgreSQL 9.5 dans avec vos applications et répondez aux questions suivantes :

  • Constatez-vous des améliorations de performances ?
  • Est-ce les changements effectués provoquent des erreurs sur vos plate-forme ?
  • Le format des journaux de transation a changé (compression). Est-ce que vos système de failover et de PRA fonctionnent comme prévu ?
  • Est-ce que la sécurité au niveu ligne (RLS) protège vos données correctement ?

La qualité des tests est essentielle dans le processus de développement de PostgreSQL. Des paquets sont disponibles pour la plupart des distributions et une image Docker a été préparée pour simplifier les tests. (voir lien ci-dessous)

Objectif fin 2015

Cette première version est désignée comme “alpha” car des changements importants sont encore possible. La première version est “beta” est prévue pour le mois d’aout et la version stable est attendue pour la fin 2015.

Liens

jeudi 9 juillet 2015 à 10h14

mercredi 8 juillet 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 6 juillet 2015

Le programme de la CommitFest pour le cycle de la 9.6 s'articule ainsi :
CF1: 1 au 31 juillet 2015
CF2: 1 au 30 septembre 2015
CF3: 1 au 30 novembre 2015
CF4: 2 au 31 janvier 2016
CF5: 1 au 31 mars 2016, pour finir dans les temps (patch rejeté par mort subite)
Feature Freeze (committer freeze): 15 avril.
https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Meeting#9.6_Schedule

PostgreSQL 9.5 Alpha 1 disponible. À vos tests ! http://www.postgresql.org/about/news/1595/

PostgreSQL Conference Europe 2015 aura lieu du 27 au 30 octobre au Vienna Marriott Hotel à Vienne (Autriche). L'appel à conférenciers est lancé jusqu'au 7 août : http://2015.pgconf.eu/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juillet

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150707033145.GA22974@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Heikki Linnakangas pushed:

  • Also trigger restartpoints based on max_wal_size on standby. When archive recovery and restartpoints were initially introduced, checkpoint_segments was ignored on the grounds that the files restored from archive don't consume any space in the recovery server. That was changed in later releases, but even then it was arguably a feature rather than a bug, as performing restartpoints as often as checkpoints during normal operation might be excessive, but you might nevertheless not want to waste a lot of space for pre-allocated WAL by setting checkpoint_segments to a high value. But now that we have separate min_wal_size and max_wal_size settings, you can bound WAL usage with max_wal_size, and still avoid consuming excessive space usage by setting min_wal_size to a lower value, so that argument is moot. There are still some issues with actually limiting the space usage to max_wal_size: restartpoints in recovery can only start after seeing the checkpoint record, while a checkpoint starts flushing buffers as soon as the redo-pointer is set. Restartpoint is paced to happen at the same leisurily speed, determined by checkpoint_completion_target, as checkpoints, but because they are started later, max_wal_size can be exceeded by upto one checkpoint cycle's worth of WAL, depending on checkpoint_completion_target. But that seems better than not trying at all, and max_wal_size is a soft limit anyway. The documentation already claimed that max_wal_size is obeyed in recovery, so this just fixes the behaviour to match the docs. However, add some weasel-words there to mention that max_wal_size may well be exceeded by some amount in recovery. http://git.postgresql.org/pg/commitdiff/d661532e27b34e9c89d0700c6ce246731e70072c
  • Initialize GIN metapage correctly when replaying metapage-update WAL record. I broke this with my WAL format refactoring patch. Before that, the metapage was read from disk, and modified in-place regardless of the LSN. That was always a bit silly, as there's no need to read the old page version from disk disk when we're overwriting it anyway. So that was changed in 9.5, but I failed to add a GinInitPage call to initialize the page-headers correctly. Usually you wouldn't notice, because the metapage is already in the page cache and is not zeroed. One way to reproduce this is to perform a VACUUM on an already vacuumed table (so that the vacuum has no real work to do), immediately after a checkpoint, and then perform an immediate shutdown. After recovery, the page headers of the metapage will be incorrectly all-zeroes. Reported by Jeff Janes http://git.postgresql.org/pg/commitdiff/47fe4d25d57c81b9d7b2ac88783a12ee487db220
  • Don't call PageGetSpecialPointer() on page until it's been initialized. After calling XLogInitBufferForRedo(), the page might be all-zeros if it was not in page cache already. btree_xlog_unlink_page initialized the page correctly, but it called PageGetSpecialPointer before initializing it, which would lead to a corrupt page at WAL replay, if the unlinked page is not in page cache. Backpatch to 9.4, the bug came with the rewrite of B-tree page deletion. http://git.postgresql.org/pg/commitdiff/fdf28853ae6a397497b79fea69f89f4f7b9aa991
  • Add assertion to check the special size is sane before dereferencing it. This seems useful to catch errors of the sort I just fixed, where PageGetSpecialPointer is called before initializing the page. http://git.postgresql.org/pg/commitdiff/302ac7f27197855afa8c89fae36c85c124ae156b
  • Use American spelling for "behavior". For consistency with the rest of the docs. Michael Paquier http://git.postgresql.org/pg/commitdiff/5b1b6bf49b44f9b26f0c9eb6d97e84973e71a0ae
  • Fix name of argument to pg_stat_file. It's called "missing_ok" in the docs and in the C code. I refrained from doing a catversion bump for this, because the name of an input argument is just documentation, it has no effect on any callers. Michael Paquier http://git.postgresql.org/pg/commitdiff/7931622d1d942dbbba8d0eab77f18f69f1ce5de0
  • Use appendStringInfoString/Char et al where appropriate. Patch by David Rowley. Backpatch to 9.5, as some of the calls were new in 9.5, and keeping the code in sync with master makes future backpatching easier. http://git.postgresql.org/pg/commitdiff/f92d6a540ac443f85f0929b284edff67da14687a
  • Don't emit a spurious space at end of line in pg_dump of event triggers. Backpatch to 9.3 and above, where event triggers were added. http://git.postgresql.org/pg/commitdiff/7b156c1e0746a46d083d7dbcd28afb303b3484ef
  • Replace obsolete autoconf macros with their modern replacements. AC_TRY_COMPILE(...) -> AC_COMPILE_IFELSE([AC_LANG_PROGRAM(...)]) AC_TRY_LINK(...) -> AC_LINK_IFELSE([AC_LANG_PROGRAM(...)]) AC_TRY_RUN(...) -> AC_RUN_IFELSE([AC_LANG_PROGRAM(...)]) AC_LANG_SAVE/RESTORE -> AC_LANG_PUSH/POP AC_DECL_SYS_SIGLIST -> AC_CHECK_DECLS(...) (per snippet in autoconf manual) Also use AC_LANG_SOURCE instead of AC_LANG_PROGRAM, where the main() function is not needed. With these changes, autoconf -Wall doesn't complain anymore. Andreas Karlsson http://git.postgresql.org/pg/commitdiff/a2edb023d08778c3346bbbf4ca82ef7f6e9283eb
  • Plug some trivial memory leaks in pg_dump and pg_upgrade. There's no point in trying to free every small allocation in these programs that are used in a one-shot fashion, but these ones seems like an improvement on readability grounds. Michael Paquier, per Coverity report. http://git.postgresql.org/pg/commitdiff/f712289ffad7c3fb6eb3be4f81adb0aa0981c9f7
  • Remove "const" from convertTSFunction()'s return type. There's no particular reason to mark it as such. The other convert* functions have no const either. http://git.postgresql.org/pg/commitdiff/a3fd7afe3090a5098f93408d47da70b47fb59e7b
  • Remove obsolete heap_formtuple/modifytuple/deformtuple functions. These variants used the old-style 'n'/' ' NULL indicators. The new-style functions have been available since version 8.1. That should be long enough that if there is still any old external code using these functions, they can just switch to the new functions without worrying about backwards compatibility. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/726117243022178e72966cbffdfb9147ec6dbbcc
  • Lift the limitation that # of clients must be a multiple of # of threads. Fabien Coelho http://git.postgresql.org/pg/commitdiff/ba3deeefb0fb9e8810b454bc7b41f27965c24aa8
  • Fix pgbench progress report behaviour when pgbench or a query gets stuck. There were two issues here. First, if a query got stuck so that it took e.g. 5 seconds, and progress interval was 1 second, no progress reports were printed until the query returned. Fix so that we wake up specifically to print the progress report. Secondly, if pgbench got stuck so that it would nevertheless not print a progress report on time, and enough time passes that it's already time to print the next progress report, just skip the one that was missed. Before this patch, it would print the missed one with 0 TPS immediately after the previous one. Fabien Coelho. Backpatch to 9.4, where progress reports were added. http://git.postgresql.org/pg/commitdiff/9031ff91a110f29e8bd4b74ddf2b5ced3ecbeaf4
  • Remove thread-emulation support from pgbench. You can no longer use pgbench with multiple threads when compiled without --enable-thread-safety. That's an acceptable limitation these days; it still works fine with -j1, and all modern platforms support threads anyway. This makes future maintenance and development of the code easier. Fabien Coelho http://git.postgresql.org/pg/commitdiff/1bc90f7a7b7441a88e2c6d4a0e9b6f9c1499ad30
  • Don't set SO_SNDBUF on recent Windows versions that have a bigger default. It's unnecessary to set it if the default is higher in the first place. Furthermore, setting SO_SNDBUF disables the so-called "dynamic send buffering" feature, which hurts performance further. This can be seen especially when the network between the client and the server has high latency. Chen Huajun http://git.postgresql.org/pg/commitdiff/4f33621f3f50286e607a3cdcc1f7a7d51075af95
  • Call getsockopt() on the correct socket. We're interested in the buffer size of the socket that's connected to the client, not the one that's listening for new connections. It happened to work, as default buffer size is the same on both, but it was clearly not wrong. Spotted by Tom Lane http://git.postgresql.org/pg/commitdiff/8e33fc1784cbd657a7238ab5639ee1f8f54a3ec0
  • Turn install.bat into a pure one line wrapper fort he perl script. Build.bat and vcregress.bat got similar treatment years ago. I'm not sure why install.bat wasn't treated at the same time, but it seems like a good idea anyway. The immediate problem with the old install.bat was that it had quoting issues, and wouldn't work if the target directory's name contained spaces. This fixes that problem. http://git.postgresql.org/pg/commitdiff/6c534fd68568452adcc9ccecb557eff74f6f0f4d

Tom Lane pushed:

  • Improve design and implementation of pg_file_settings view. As first committed, this view reported on the file contents as they were at the last SIGHUP event. That's not as useful as reporting on the current contents, and what's more, it didn't work right on Windows unless the current session had serviced at least one SIGHUP. Therefore, arrange to re-read the files when pg_show_all_settings() is called. This requires only minor refactoring so that we can pass changeVal = false to set_config_option() so that it won't actually apply any changes locally. In addition, add error reporting so that errors that would prevent the configuration files from being loaded, or would prevent individual settings from being applied, are visible directly in the view. This makes the view usable for pre-testing whether edits made in the config files will have the desired effect, before one actually issues a SIGHUP. I also added an "applied" column so that it's easy to identify entries that are superseded by later entries; this was the main use-case for the original design, but it seemed unnecessarily hard to use for that. Also fix a 9.4.1 regression that allowed multiple entries for a PGC_POSTMASTER variable to cause bogus complaints in the postmaster log. (The issue here was that commit bf007a27acd7b2fb unintentionally reverted 3e3f65973a3c94a6, which suppressed any duplicate entries within ParseConfigFp. However, since the original coding of the pg_file_settings view depended on such suppression *not* happening, we couldn't have fixed this issue now without first doing something with pg_file_settings. Now we suppress duplicates by marking them "ignored" within ProcessConfigFileInternal, which doesn't hide them in the view.) Lesser changes include: Drive the view directly off the ConfigVariable list, instead of making a basically-equivalent second copy of the data. There's no longer any need to hang onto the data permanently, anyway. Convert show_all_file_settings() to do its work in one call and return a tuplestore; this avoids risks associated with assuming that the GUC state will hold still over the course of query execution. (I think there were probably latent bugs here, though you might need something like a cursor on the view to expose them.) Arrange to run SIGHUP processing in a short-lived memory context, to forestall process-lifespan memory leaks. (There is one known leak in this code, in ProcessConfigDirectory; it seems minor enough to not be worth back-patching a specific fix for.) Remove mistaken assignment to ConfigFileLineno that caused line counting after an include_dir directive to be completely wrong. Add missed failure check in AlterSystemSetConfigFile(). We don't really expect ParseConfigFp() to fail, but that's not an excuse for not checking. http://git.postgresql.org/pg/commitdiff/62d16c7fc5614d9f4d0dd1a9f164b232c273c128
  • Run the C portions of guc-file.l through pgindent. Yeah, I know, pretty anal-retentive of me. But we oughta find some way to automate this for the .y and .l files. http://git.postgresql.org/pg/commitdiff/2bdc51a2946f9a66688eb705cd0cb584ebd8240a
  • Code + docs review for escaping of option values (commit 11a020eb6). Avoid memory leak from incorrect choice of how to free a StringInfo (resetStringInfo doesn't do it). Now that pg_split_opts doesn't scribble on the optstr, mark that as "const" for clarity. Attach the commentary in protocol.sgml to the right place, and add documentation about the user-visible effects of this change on postgres' -o option and libpq's PGOPTIONS option. http://git.postgresql.org/pg/commitdiff/cbc8d65639344c390a1d1a7f646c186ff3ad8693
  • Desultory review of 9.5 release notes. Minor corrections and clarifications. Notably, for stuff that got moved out of contrib, make sure it's documented somewhere other than "Additional Modules". I'm sure these need more work, but that's all I have time for today. http://git.postgresql.org/pg/commitdiff/85c25fdbd7d624928bb8a1f1fd1e5043ec35dd74
  • Stamp 9.5alpha1. http://git.postgresql.org/pg/commitdiff/f78329d594c2fe893f9174d5b3da7d3fbc6dd8b6
  • Remove useless check for NULL subexpression. Coverity rightly gripes that it's silly to have a test here when the adjacent ExecEvalExpr() would choke on a NULL expression pointer. Petr Jelinek http://git.postgresql.org/pg/commitdiff/131926a52da0fbd77678cbd887914c83b48faa2d
  • Stamp HEAD as 9.6devel. Let the hacking begin ... http://git.postgresql.org/pg/commitdiff/cf8d65de10ebd026f6534eeb275cae46d7bffb4f
  • Stamp shared-library minor version numbers for 9.6. http://git.postgresql.org/pg/commitdiff/019f7813da69da05484faa63b6a29a8df773d19b
  • Fix broken link in documentation. HP's web server has apparently become case-sensitive sometime recently. Per bug #13479 from Daniel Abraham. Corrected link identified by Alvaro. http://git.postgresql.org/pg/commitdiff/7f32dbcd73b9a75d09db386fa81c31f42e6f0d3a
  • Make sampler_random_fract() actually obey its API contract. This function is documented to return a value in the range (0,1), which is what its predecessor anl_random_fract() did. However, the new version depends on pg_erand48() which returns a value in [0,1). The possibility of returning zero creates hazards of division by zero or trying to compute log(0) at some call sites, and it might well break third-party modules using anl_random_fract() too. So let's change it to never return zero. Spotted by Coverity. Michael Paquier, cosmetically adjusted by me http://git.postgresql.org/pg/commitdiff/d7c19d68550eb6018e8581a73a351905f4cc435c
  • Don't leave pg_hba and pg_ident data lying around in running backends. Free the contexts holding this data after we're done using it, by the expedient of attaching them to the PostmasterContext which we were already taking care to delete (and where, indeed, this data used to live before commits e5e2fc842c418432 and 7c45e3a3c682f855). This saves a probably-usually-negligible amount of space per running backend. It also avoids leaving potentially-security-sensitive data lying around in memory in processes that don't need it. You'd have to be unusually paranoid to think that that amounts to a live security bug, so I've not gone so far as to forcibly zero the memory; but there surely isn't a good reason to keep this data around. Arguably this is a memory management bug in the aforementioned commits, but it doesn't seem important enough to back-patch. http://git.postgresql.org/pg/commitdiff/1e24cf645d24aab3ea39a9d259897fd0cae4e4b6
  • Add an optional missing_ok argument to SQL function current_setting(). This allows convenient checking for existence of a GUC from SQL, which is particularly useful when dealing with custom variables. David Christensen, reviewed by Jeevan Chalke http://git.postgresql.org/pg/commitdiff/10fb48d66de76e7dc1e36ef18af502ed9600352f
  • Fix misuse of TextDatumGetCString(). "TextDatumGetCString(PG_GETARG_TEXT_P(x))" is formally wrong: a text* is not a Datum. Although this coding will accidentally fail to fail on all known platforms, it risks leaking memory if a detoast step is needed, unlike "TextDatumGetCString(PG_GETARG_DATUM(x))" which is what's used elsewhere. Make pg_get_object_address() fall in line with other uses. Noted while reviewing two-arg current_setting() patch. http://git.postgresql.org/pg/commitdiff/ac50f84866b22f239025bf37c9c7492cc4ce2dfd
  • Make numeric form of PG version number readily available in Makefiles. Expose PG_VERSION_NUM (e.g., "90600") as a Make variable; but for consistency with the other Make variables holding similar info, call the variable just VERSION_NUM not PG_VERSION_NUM. There was some discussion of making this value available as a pg_config value as well. However, that would entail substantially more work than this two-line patch. Given that there was not exactly universal consensus that we need this at all, let's just do a minimal amount of work for now. Michael Paquier, reviewed by Pavel Stehule http://git.postgresql.org/pg/commitdiff/a5d489ccb7e613c7ca3be6141092b8c1d2c13fa7
  • Improve pg_restore's -t switch to match all types of relations. -t will now match views, foreign tables, materialized views, and sequences, not only plain tables. This is more useful, and also more consistent with the behavior of pg_dump's -t switch, which has always matched all relation types. We're still not there on matching pg_dump's behavior entirely, so mention that in the docs. Craig Ringer, reviewed by Pavel Stehule http://git.postgresql.org/pg/commitdiff/5671aaca87c47128f6a1e0556ce9c7512096ad87
  • Add psql \ev and \sv commands for editing and showing view definitions. These are basically just like the \ef and \sf commands for functions. Petr Korobeinikov, reviewed by Jeevan Chalke, some changes by me http://git.postgresql.org/pg/commitdiff/8eb6407aaeb6cbd972839e356b436bb698f51cff
  • Add documentation and regression tests concerning rounding of numerics. Michael Paquier, reviewed by Fabien Coelho http://git.postgresql.org/pg/commitdiff/5e7c3d91bf24a212b42c912234c6cb37d75e0292
  • Fix bad grammar in brin.sgml. Christoph Berg http://git.postgresql.org/pg/commitdiff/252404625aa98fa5f93a45a8dcffdc179981820a
  • Further reduce overhead for passing plpgsql variables to the executor. This builds on commit 21dcda2713656a7483e3280ac9d2ada20a87a9a9 by keeping a plpgsql function's shared ParamListInfo's entries for simple variables (PLPGSQL_DTYPE_VARs) valid at all times. That adds a few cycles to each assignment to such variables, but saves significantly more cycles each time they are used; so except in the pathological case of many dead stores, this should always be a win. Initial testing says it's good for about a 10% speedup of simple calculations; more in large functions with many datums. We can't use this method for row/record references unfortunately, so what we do for those is reset those ParamListInfo slots after use; which we can skip doing unless some of them were actually evaluated during the previous evaluation call. So this should frequently be a win as well, while worst case is that it's similar cost to the previous approach. Also, closer study suggests that the previous method of instantiating a new ParamListInfo array per evaluation is actually probably optimal for cursor-opening executor calls. The reason is that whatever is visible in the array is going to get copied into the cursor portal via copyParamList. So if we used the function's main ParamListInfo for those calls, we'd end up with all of its DTYPE_VAR vars getting copied, which might well include large pass-by-reference values that the cursor actually has no need for. To avoid a possible net degradation in cursor cases, go back to creating and filling a private ParamListInfo in those cases (which therefore will be exactly the same speed as before 21dcda271365). We still get some benefit out of this though, because this approach means that we only have to defend against copyParamList's try-to-fetch-every-slot behavior in the case of an unshared ParamListInfo; so plpgsql_param_fetch() can skip testing expr->paramnos in the common case. To ensure that the main ParamListInfo's image of a DTYPE_VAR datum is always valid, all assignments to such variables are now funneled through assign_simple_var(). But this makes for cleaner and shorter code anyway. http://git.postgresql.org/pg/commitdiff/6c82d8d1fdb1f1265f93d89640edcbd0ae22c627
  • Fix some typos in regression test comments. Back-patch to avoid unnecessary cross-branch differences. CharSyam http://git.postgresql.org/pg/commitdiff/551654977022097ac408b483b3be9887a99f0ce0
  • Make a editorial pass over pgbench's error messages. The lack of consistency, and lack of attention to our message style guidelines, was a bit striking. Try to make 'em better. http://git.postgresql.org/pg/commitdiff/22ba5563adacd162d97ff3c80eac4893574f1e17

Peter Eisentraut pushed:

Andres Freund pushed:

Robert Haas pushed:

Ãlvaro Herrera pushed:

Fujii Masao pushed:

  • Make XLogFileCopy() look the same as in 9.4. XLogFileCopy() was changed heavily in commit de76884. However it was partially reverted in commit 7abc685 and most of those changes to XLogFileCopy() were no longer needed. Then commit 7cbee7c removed those unnecessary code, but XLogFileCopy() looked different in master and 9.4 though the contents are almost the same. This patch makes XLogFileCopy() look the same in master and back-branches, which makes back-patching easier, per discussion on pgsql-hackers. Back-patch to 9.5. Discussion: 55760844.7090703@iki.fi Michael Paquier http://git.postgresql.org/pg/commitdiff/8217370864c950ea28c7f940442fe48c701461c2
  • Make use of xlog_internal.h's macros in WAL-related utilities. Commit 179cdd09 added macros to check if a filename is a WAL segment or other such file. However there were still some instances of the strlen + strspn combination to check for that in WAL-related utilities like pg_archivecleanup. Those checks can be replaced with the macros. This patch makes use of the macros in those utilities and which would make the code a bit easier to read. Back-patch to 9.5. Michael Paquier http://git.postgresql.org/pg/commitdiff/fb174687f7a730edcf301949785a6ac0dbfd70d0
  • Make WAL-related utilities handle .partial WAL files properly. Commit de76884 changed an archive recovery so that the last WAL segment with old timeline was renamed with suffix .partial. It should have updated WAL-related utilities so that they can handle such .paritial WAL files, but we forgot that. This patch changes pg_archivecleanup so that it can clean up even archived WAL files with .partial suffix. Also it allows us to specify .partial WAL file name as the command-line argument "oldestkeptwalfile". This patch also changes pg_resetxlog so that it can remove .partial WAL files in pg_xlog directory. pg_xlogdump cannot handle .partial WAL files. Per discussion, we decided only to document that limitation instead of adding the fix. Because a user can easily work around the limitation (i.e., just remove .partial suffix from the file name) and the fix seems complicated for very narrow use case. Back-patch to 9.5 where the problem existed. Review by Michael Paquier. Discussion: http://www.postgresql.org/message-id/CAHGQGwGxMKnVHGgTfiig2Bt_2djec0in3-DLJmtg7+nEiidFdQ@mail.gmail.com http://git.postgresql.org/pg/commitdiff/8650d161ae4369ec64a6fc7b7cbd0e6e55c3a7aa
  • Remove incorrect warning from pg_archivecleanup document. The .backup file name can be passed to pg_archivecleanup even if it includes the extension which is specified in -x option. However, previously the document incorrectly warned a user not to do that. Back-patch to 9.2 where pg_archivecleanup's -x option and the warning were added. http://git.postgresql.org/pg/commitdiff/906e9249494bda975914b78fdd743609e1f83016

Andrew Dunstan pushed:

Joe Conway pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Amit Langote sent in a patch to adjust the errorcode for the error related to checking the flag bgw_flags in BackgroundWorkerInitializeConnection*() functions so that it matches the treatment in SanityCheckBackgroundWorker().

Jeff Janes sent in a patch to that implements the vm scan for truncation.

Michael Paquier and Heikki Linnakangas traded patches to fix some infelicities exposed by pg_rewind.

Amit Kapila sent in a patch to reduce ProcArrayLock contention by adding a mechanism to GroupClear the Xid during ProcArrayEndTransaction().

Jeff Janes sent in a patch to make a crash scenario he'd found easier to reproduce.

Franck Verrot sent in a patch to mention column name in error messages.

Peter Geoghegan sent in a patch to fix a bug in bttext_abbrev_convert().

Andres Freund sent in a patch to rework the way multixact truncations work.

Michael Paquier sent in a patch to add a couple of return value checks that were missing for JsonbIteratorNext in jsonfuncs.c.

Michael Paquier sent in a patch to add missing checks on return value of timestamp2tm in datetime.c.

Amit Kapila sent in a patch to extend pg_stat_activity to include wait_event.

Michael Paquier and Petr Jelinek traded fix a possible NULL pointer dereferernce in tablesample.c

Jeff Janes sent in another revision of a patch to update pg_trgm to use more modern facilities for a speed boost.

Michael Paquier sent in a patch to remove an unneeded NULL-pointer check in FreeSpaceMapTruncateRel.

Peter Eisentraut sent in a patch to allow the -X stream method to specify a replication slot and create it in the same run if needed.

Simon Riggs sent in a patch to reduce ClogControlLock contention at commit time.

Rahila Syed sent in a POC patch to allow tracking VACUUM progress.

Amit Kapila sent in two more revisions of a patch to rename mapfile if backupfile not present.

SAWADA Masahiko sent in three more revisions of a patch to add a "frozen" bit to the visibility map.

Kyotaro HORIGUCHI sent in another revision of a patch to allow asynchronous execution on a FDW.

Etsuro Fujita sent in a patch to fix an issue it's possible that the result of SELECT ... ORDER BY ... FOR UPDATE is not sorted correctly due to concurrent updates' having replaced the sort key columns with new values.

Robbie Harwood sent in a patch to add GSSAPI encryption support.

Amit Kapila sent in another revision of a patch to allow assessing parallel-safety.

Fabien COELHO sent in another revision of a patch to allow backslash-continuations in custom pgbench scripts.

Peter Geoghegan sent in a patch to make sorting text under a non-C locale faster than in 9.5.

Noah Misch sent in a patch to fix some strxfrm overflow bugs.

Michael Paquier sent in two revisions of a patch to fix an issue where table constraints and other objects were not being persisted over ALTERs.

Jan de Visser sent in another revision of a patch to let pg_ctl check the result of a postmaster config reload.

Amit Kapila sent in another revision of a patch to implement parallel seq scan.

Noah Misch sent in two revisions of a patch to implement xlc atomics.

Marco Atzeri sent in a patch to fix a 9.5alpha1 build failure with Perl 5.22.

Julien Rouhaud sent in a patch to allow using the PID of the backend in psql.

David Rowley sent in a patch to fix some infelicities in an as yet uncommitted memory accounting patch.

Joe Conway and Michael Paquier traded patches to add polymorphic functions to dblink.

Uriy Zhuravlev sent in another revision of a patch to enhance ALTER OPERATOR to allow setting some of the operator's parameters.

David Christensen sent in a patch to add \ddp to the tab completions in psql.

Pavel Stehule sent in another revision of a patch to allow raw output from COPY.

par N Bougain le mercredi 8 juillet 2015 à 07h01

lundi 29 juin 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 28 juin 2015

La PGConf.DE se tiendra à Hamburg (Allemagne) les 26 et 27 novembre au Lindner Hotel am Michel. L'appel à conférenciers court jusqu'au 13 septembre 2015 : http://2015.pgconf.de/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juin

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150628215355.GB30019@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Noah Misch pushed:

  • Truncate strings in tarCreateHeader() with strlcpy(), not sprintf(). This supplements the GNU libc bug #6530 workarounds introduced in commit 54cd4f04576833abc394e131288bf3dd7dcf4806. On affected systems, a tar-format pg_basebackup failed when some filename beneath the data directory was not valid character data in the postmaster/walsender locale. Back-patch to 9.1, where pg_basebackup was introduced. Extant, bug-prone conversion specifications receive only ASCII bytes or involve low-importance messages. http://git.postgresql.org/pg/commitdiff/4318118edd5582696027f357771e0a8b091fe2bf

Robert Haas pushed:

Tom Lane pushed:

  • Improve inheritance_planner()'s performance for large inheritance sets. Commit c03ad5602f529787968fa3201b35c119bbc6d782 introduced a planner performance regression for UPDATE/DELETE on large inheritance sets. It required copying the append_rel_list (which is of size proportional to the number of inherited tables) once for each inherited table, thus resulting in O(N^2) time and memory consumption. While it's difficult to avoid that in general, the extra work only has to be done for append_rel_list entries that actually reference subquery RTEs, which inheritance-set entries will not. So we can buy back essentially all of the loss in cases without subqueries in FROM; and even for those, the added work is mainly proportional to the number of UNION ALL subqueries. Back-patch to 9.2, like the previous commit. Tom Lane and Dean Rasheed, per a complaint from Thomas Munro. http://git.postgresql.org/pg/commitdiff/2cb9ec1bcb35dd6b4cf7a4a325aaa9791444e69d
  • Docs: fix claim that to_char('FM') removes trailing zeroes. Of course, what it removes is leading zeroes. Seems to have been a thinko in commit ffe92d15d53625d5ae0c23f4e1984ed43614a33d. Noted by Hubert Depesz Lubaczewski. http://git.postgresql.org/pg/commitdiff/d759b7eb6aee12bd52516905d790072845b4356f
  • Fix the logic for putting relations into the relcache init file. Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index into the relcache init file, because that index is not used by any syscache. However, we have historically nailed that index into cache for performance reasons. The upshot was that load_relcache_init_file always decided that the init file was busted and silently ignored it, resulting in a significant hit to backend startup speed. To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around RelationSupportsSysCache(), which can know about additional relations that should be in the init file despite being unknown to syscache.c. Also install some guards against future mistakes of this type: make write_relcache_init_file Assert that all nailed relations get written to the init file, and make load_relcache_init_file emit a WARNING if it takes the "wrong number of nailed relations" exit path. Now that we remove the init files during postmaster startup, that case should never occur in the field, even if we are starting a minor-version update that added or removed rels from the nailed set. So the warning shouldn't ever be seen by end users, but it will show up in the regression tests if somebody breaks this logic. Back-patch to all supported branches, like the previous commit. http://git.postgresql.org/pg/commitdiff/5d1ff6bd559ea8df1b7302e245e690b01b9a4fa4
  • Avoid passing NULL to memcmp() in lookups of zero-argument functions. A few places assumed they could pass NULL for the argtypes array when looking up functions known to have zero arguments. At first glance it seems that this should be safe enough, since memcmp() is surely not allowed to fetch any bytes if its count argument is zero. However, close reading of the C standard says that such calls have undefined behavior, so we'd probably best avoid it. Since the number of places doing this is quite small, and some other places looking up zero-argument functions were already passing dummy arrays, let's standardize on the latter solution rather than hacking the function lookup code to avoid calling memcmp() in these cases. I also added Asserts to catch any future violations of the new rule. Given the utter lack of any evidence that this actually causes any problems in the field, I don't feel a need to back-patch this change. Per report from Piotr Stefaniak, though this is not his patch. http://git.postgresql.org/pg/commitdiff/0a52d378b03b7d5ab1d64627a87edaf5ed311c6c

Peter Eisentraut pushed:

Heikki Linnakangas pushed:

  • Add missing newline to debug-message. Michael Paquier http://git.postgresql.org/pg/commitdiff/9cb36981fbbf2f298db2476101f4475c52d00fbb
  • Fix a couple of bugs with wal_log_hints. 1. Replay of the WAL record for setting a bit in the visibility map contained an assertion that a full-page image of that record type can only occur with checksums enabled. But it can also happen with wal_log_hints, so remove the assertion. Unlike checksums, wal_log_hints can be changed on the fly, so it would be complicated to figure out if it was enabled at the time that the WAL record was generated. 2. wal_log_hints has the same effect on the locking needed to read the LSN of a page as data checksums. BufferGetLSNAtomic() didn't get the memo. Backpatch to 9.4, where wal_log_hints was added. http://git.postgresql.org/pg/commitdiff/4b8e24b9ad308c30dbe2184e06848e638e018114
  • Fix typo in comment. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/7845db2aa778aa751b41cff72c41c94993e975e3
  • Add missing_ok option to the SQL functions for reading files. This makes it possible to use the functions without getting errors, if there is a chance that the file might be removed or renamed concurrently. pg_rewind needs to do just that, although this could be useful for other purposes too. (The changes to pg_rewind to use these functions will come in a separate commit.) The read_binary_file() function isn't very well-suited for extensions.c's purposes anymore, if it ever was. So bite the bullet and make a copy of it in extension.c, tailored for that use case. This seems better than the accidental code reuse, even if it's a some more lines of code. Michael Paquier, with plenty of kibitzing by me. http://git.postgresql.org/pg/commitdiff/cb2acb1081e13b4b27a76c6b5311115528e49c59
  • Don't choke on files that are removed while pg_rewind runs. If a file is removed from the source server, while pg_rewind is running, the invocation of pg_read_binary_file() will fail. Use the just-added missing_ok option to that function, to have it return NULL instead, and handle that gracefully. And similarly for pg_ls_dir and pg_stat_file. Reported by Fujii Masao, fix by Michael Paquier. http://git.postgresql.org/pg/commitdiff/b36805f3c54fe0e50e58bb9e6dad66daca46fbf6
  • Fix double-XLogBeginInsert call in GIN page splits. If data checksums or wal_log_hints is on, and a GIN page is split, the code to find a new, empty, block was called after having already called XLogBeginInsert(). That causes an assertion failure or PANIC, if finding the new block involves updating a FSM page that had not been modified since last checkpoint, because that update is WAL-logged, which calls XLogBeginInsert again. Nested XLogBeginInsert calls are not supported. To fix, rearrange GIN code so that XLogBeginInsert is called later, after finding the victim buffers. Reported by Jeff Janes. http://git.postgresql.org/pg/commitdiff/a45c70acf35e43257d86313dcbb7bb0e5201fab1
  • Promote the assertion that XLogBeginInsert() is not called twice into ERROR. Seems like cheap insurance for WAL bugs. A spurious call to XLogBeginInsert() in itself would be fairly harmless, but if there is any data registered and the insertion is not completed/cancelled properly, there is a risk that the data ends up in a wrong WAL record. Per Jeff Janes's suggestion. http://git.postgresql.org/pg/commitdiff/a32c3ec893cafbd3a4b42c34270a80198f28f123
  • Fix markup in docs. Oops. I could swear I built the docs before pushing, but I guess not.. http://git.postgresql.org/pg/commitdiff/6ab4d38ab085b0177d7ce63f7e1f2fb3f3a8e4a5

Fujii Masao pushed:

Andres Freund pushed:

  • Fix the fallback memory barrier implementation to be reentrant. This was essentially "broken" since 0c8eda62; but until more recently (14e8803f) barriers usage in signal handlers was infrequent. The failure to be reentrant was noticed because the test_shm_mq, which uses memory barriers at a high frequency, occasionally got stuck on some solaris buildfarm animals. Turns out, those machines use sun studio 12.1, which doesn't yet have efficient memory barrier support. A machine with a newer sun studio did not fail. Forcing the barrier fallback to be used on x86 allows to reproduce the problem. The new fallback is to use kill(PostmasterPid, 0) based on the theory that that'll always imply a barrier due to checking the liveliness of PostmasterPid on systems old enough to need fallback support. It's hard to come up with a good and performant fallback. I'm not backpatching this for now - the problem isn't active in the back branches, and we haven't backpatched barrier changes for now. Additionally master looks entirely different than the back branches due to the new atomics abstraction. It seems better to let this rest in master, where the non-reentrancy actively causes a problem, and then consider backpatching. Found-By: Robert Haas Discussion: 55626265.3060800@dunslane.net http://git.postgresql.org/pg/commitdiff/1b468a131bd260c9041484f78b8580c7f232d580
  • Fix test_decoding's handling of nonexistant columns in old tuple versions. test_decoding used fastgetattr() to extract column values. That's wrong when decoding updates and deletes if a table's replica identity is set to FULL and new columns have been added since the old version of the tuple was created. Due to the lack of a crosscheck with the datum's natts values an invalid value will be output, leading to errors or worse. Bug: #13470 Reported-By: Krzysztof Kotlarski Discussion: 20150626100333.3874.90852@wrigleys.postgresql.org Backpatch to 9.4, where the feature, including the bug, was added. http://git.postgresql.org/pg/commitdiff/d47a1136e441cebe7ae7fe72d70eb8ce278d5cd6

Ãlvaro Herrera pushed:

Simon Riggs pushed:

Kevin Grittner pushed:

  • Add opaque declaration of HTAB to tqual.h. Commit b89e151054a05f0f6d356ca52e3b725dd0505e53 added the ResolveCminCmaxDuringDecoding declaration to tqual.h, which uses an HTAB parameter, without declaring HTAB. It accidentally fails to fail to build with current sources because a declaration happens to be included, directly or indirectly, in all source files that currently use tqual.h before tqual.h is first included, but we shouldn't count on that. Since an opaque declaration is enough here, just use that, as was done in snapmgr.h. Backpatch to 9.4, where the HTAB reference was added to tqual.h. http://git.postgresql.org/pg/commitdiff/604e99396de02f6f23950ee373c13335d2ccdf05
  • Fix comment for GetCurrentIntegerTimestamp(). The unit of measure is microseconds, not milliseconds. Backpatch to 9.3 where the function and its comment were added. http://git.postgresql.org/pg/commitdiff/cca8ba9529f8815acd23fe88c32763765d0e1b68

Tatsuo Ishii pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Peter Geoghegan sent in a patch to allow JSON[B] arrays to take negative subscripts.

Tomas Vondra sent in a patch to add density correction to sampling for statistics collection.

Michael Paquier sent in a patch to fix some white space in pg_rewind error messages.

Michael Paquier and Robert Haas traded patches to fix an issue where a dangerous rm -rf was being issued in the global makefile.

Michael Paquier sent in two more revisions of a patch to update the hash index creation warning.

Abhijit Menon-Sen sent in a patch to introduce XLogLockBlockRangeForCleanup().

Fabien COELHO sent in another revision of a patch to add checkpointer continuous flushing.

Michael Paquier sent in another revision of a patch to add support for TAP tests on Windows.

Michael Paquier sent in a patch to improve log capture of TAP tests and fix race conditions.

Fabrízio de Royes Mello sent in two more revisions of a patch to add CINE for ALTER TABLE ... ADD COLUMN.

Uriy Zhuravlev sent in another revision of a patch to implement ALTER OPERATOR.

Craig Ringer sent in a patch to implement ALTER TABLE ... ALTER CONSTRAINT ... SET DEFERRABLE on UNIQUE or PK.

Oskari Saarenmaa sent in a patch to add -lrt to configure for sched_yield on Solaris.

Jim Nasby sent in a patch to ensure that object_classes is properly sized.

Marco Nenciarini sent in a patch to fix an off-by-one bug which caused VACUUM FREEZE to mistakenly cancel standby sessions.

Peter Geoghegan sent in a patch to add some compatibility notes for UPSERT with foreign data wrappers.

Amit Kapila sent in a patch to improve the performance of DROP TABLE when the shared_buffers setting is high.

Jeff Janes sent in a patch to make pg_trgm perform better by supporting the triconsistent function, introduced in version 9.4 of the server, to make it faster to implement indexed queries where some keys are common and some are rare.

Tom Lane sent in a patch to refactor the way the pg_file_settings view works.

par N Bougain le lundi 29 juin 2015 à 21h21

samedi 27 juin 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 21 juin 2015

La PostgreSQL Conference Europe 2015 qui aura lieu du 27 au 30 octobre accepte à présent les inscriptions des auditeurs : http://2015.pgconf.eu/registration/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juin

PostgreSQL Local

  • La deuxième Conférence PostgreSQL Suisse se tiendra à Rapperswil les 25 & 26 juin 2015 : http://www.postgres-conference.ch/
  • La conférence PGDay UK aura lieu le 7 juillet 2015 - elle vise les membres de la communauté PostgreSQL anglaise. L'appel à conférenciers expire le 13 avril : http://www.postgresqlusergroup.org.uk
  • Le PGDay Campinas 2015 aura lieu à Campinas (Brésil) le 7 août. L'appel à conférenciers expire le 31 mai : http://pgdaycampinas.com.br/english/
  • L'appel à conférenciers pour le PostgresOpen 2015, programmé à Dallas (Texas) du 16 au 18 septembre, a été lancé : http://2015.postgresopen.org/callforpapers/
  • L'appel à conférenciers pour la session PostgreSQL n°7 (24 septembre 2015 à Paris) est lancé jusqu'au 15 juin 2015 : call-for-paper <AT> postgresql-sessions <DOT> org : http://www.postgresql-sessions.org/7/about
  • PostgreSQL Conference Europe 2015 aura lieu du 27 au 30 octobre au Vienna Marriott Hotel à Vienne (Autriche). L'appel à conférenciers est lancé jusqu'au 7 août : http://2015.pgconf.eu/
  • PGConf Silicon Valley 2015 se tiendra au centre de convention sud de San Francisco les 17 & 18 novembre. L'appel à conférenciers porte jusqu'au 15 juin : http://www.pgconfsv.com

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150621221026.GB32298@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Michael Meskes pushed:

Andrew Dunstan pushed:

Noah Misch pushed:

  • Detect setlocale(LC_CTYPE, NULL) clobbering previous return values. POSIX permits setlocale() calls to invalidate any previous setlocale() return values. Commit 5f538ad004aa00cf0881f179f0cde789aad4f47e neglected to account for that. In advance of fixing that bug, switch to failing hard on affected configurations. This is a planned temporary commit to assay buildfarm-represented configurations. http://git.postgresql.org/pg/commitdiff/b76e76be460a240e99c33f6fb470dd1d5fe01a2a
  • Revert "Detect setlocale(LC_CTYPE, NULL) clobbering previous return values." This reverts commit b76e76be460a240e99c33f6fb470dd1d5fe01a2a. The buildfarm yielded no related failures. http://git.postgresql.org/pg/commitdiff/1f2a378de41bf3e516b8d2c4d65790aeefbfb89d
  • Fix failure to copy setlocale() return value. POSIX permits setlocale() calls to invalidate any previous setlocale() return values, but commit 5f538ad004aa00cf0881f179f0cde789aad4f47e neglected to account for setlocale(LC_CTYPE, NULL) doing so. The effect was to set the LC_CTYPE environment variable to an unintended value. pg_perm_setlocale() sets this variable to assist PL/Perl; without it, Perl would undo PostgreSQL's locale settings. The known-affected configurations are 32-bit, release builds using Visual Studio 2012 or Visual Studio 2013. Visual Studio 2010 is unaffected, as were all buildfarm-attested configurations. In principle, this bug could leave the wrong LC_CTYPE in effect after PL/Perl use, which could in turn facilitate problems like corrupt tsvector datums. No known platform experiences that consequence, because PL/Perl on Windows does not use this environment variable. The bug has been user-visible, as early postmaster failure, on systems with Windows ANSI code page set to CP936 for "Chinese (Simplified, PRC)" and probably on systems using other multibyte code pages. (SetEnvironmentVariable() rejects values containing character data not valid under the Windows ANSI code page.) Back-patch to 9.4, where the faulty commit first appeared. Reported by Didi Hu and æž—é¹ç¨‹. Reviewed by Tom Lane, though this fix strategy was not his first choice. http://git.postgresql.org/pg/commitdiff/f0a264a362343287051c4737b01aa3ebe36f21a6

Robert Haas pushed:

Tom Lane pushed:

  • Fix bogus range_table_mutator() logic for RangeTblEntry.tablesample. Must make a copy of the TableSampleClause node; the previous coding modified the input data structure in-place. Petr Jelinek http://git.postgresql.org/pg/commitdiff/be87143fe90adf8862791aeddd76151e88ce5603
  • In immediate shutdown, postmaster should not exit till children are gone. This adjusts commit 82233ce7ea42d6ba519aaec63008aff49da6c7af so that the postmaster does not exit until all its child processes have exited, even if the 5-second timeout elapses and we have to send SIGKILL. There is no great value in having the postmaster process quit sooner, and doing so can mislead onlookers into thinking that the cluster is fully terminated when actually some child processes still survive. This effect might explain recent test failures on buildfarm member hamster, wherein we failed to restart a cluster just after shutting it down with "pg_ctl stop -m immediate". I also did a bit of code review/beautification, including fixing a faulty use of the Max() macro on a volatile expression. Back-patch to 9.4. In older branches, the postmaster never waited for children to exit during immediate shutdowns, and changing that would be too much of a behavioral change. http://git.postgresql.org/pg/commitdiff/48913db887e6a41fa3f1b6cdf80ee89e38f21d9d

Ãlvaro Herrera pushed:

Peter Eisentraut pushed:

Andres Freund pushed:

  • Add missing check for wal_debug GUC. 9a20a9b2 added a new elog(), enabled when WAL_DEBUG is defined. The other WAL_DEBUG dependant messages check for the wal_debug GUC, but this one did not. While at it replace 'upto' with 'up to'. Discussion: 20150610110253.GF3832@alap3.anarazel.de Backpatch to 9.4, the first release containing 9a20a9b2. http://git.postgresql.org/pg/commitdiff/90231cd5188c43da94f58f7a839eee9286d0f864
  • Improve multixact emergency autovacuum logic. Previously autovacuum was not necessarily triggered if space in the members slru got tight. The first problem was that the signalling was tied to values in the offsets slru, but members can advance much faster. Thats especially a problem if old sessions had been around that previously prevented the multixact horizon to increase. Secondly the skipping logic doesn't work if the database was restarted after autovacuum was triggered - that knowledge is not preserved across restart. This is especially a problem because it's a common panic-reaction to restart the database if it gets slow to anti-wraparound vacuums. Fix the first problem by separating the logic for members from offsets. Trigger autovacuum whenever a multixact crosses a segment boundary, as the current member offset increases in irregular values, so we can't use a simple modulo logic as for offsets. Add a stopgap for the second problem, by signalling autovacuum whenver ERRORing out because of boundaries. Discussion: 20150608163707.GD20772@alap3.anarazel.de Backpatch into 9.3, where it became more likely that multixacts wrap around. http://git.postgresql.org/pg/commitdiff/667912aee649c3608e003568e4b47d95251b1c8c

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

David Rowley sent in a patch to let the executor notice when multiple aggregates in a query share a transition function, executing the common ones only once per row.

David Rowley sent in another revision of a patch to improve some appendStringInfo* calls.

Petr Jelinek sent in a patch to extend CREATE EXTENSION to include its dependencies via the optional RECURSIVE keyword.

Fabien COELHO sent in another revision of a patch to fix pgbench --progress report under (very) low rate.

Tomas Vondra sent in a patch to teach the expression walker about RestrictInfo.

Amit Kapila sent in a patch to rename mapfile if backupfile not present.

Vik Fearing sent in a patch to add tab completion to psql for CREATE SEQUENCE.

Michael Paquier sent in a patch to reproduce a problem with pg_rewind.

Michael Paquier sent in a patch to make process_remote_file ignore files named as pg_xlog/xlogtemp.*.

SAWADA Masahiko sent in a patch to give the GIN function of pageinspect a consistent data type.

Fabien COELHO sent in another revision of a patch to smooth checkpoint flushing.

Tomas Vondra sent in another revision of a patch to make a better ndistinct estimator.

Brendan Jurd sent in two revisions of a patch to add a new built-in function pg_notify_queue_saturation().

Michael Paquier sent in three more revisions of a patch to improve the way TAP tests log their output using IPC::Run::run.

Michael Paquier sent in a patch to add a missing -w switch to the pg_ctl stop call in pg_regress.

Robert Haas, Tom Lane, and Dean Rasheed traded patches to fix some infelicities recently introduced in the table inheritance part of the planner.

Marti Raudsepp sent in a patch to fix pg_upgrade when postgres template1 aren't in the default database.

Petr Jelinek sent in another revision of a patch to implement tab completion in psql for TABLESAMPLE.

Ãlvaro Herrera sent in a patch to add pg_get_multixact_members() and pg_get_multixact_range().

Thomas Munro sent in a patch to fix an issue where the get_relation_info comment was out of sync with the code nearby.

Andres Freund sent in a patch to rework the way multixact truncations work with the goal of making them WAL logged.

Fabien COELHO sent in a patch to allow backslash-continuations in custom scripts.

par N Bougain le samedi 27 juin 2015 à 00h40

Nouvelles hebdomadaires de PostgreSQL - 14 juin 2015

Mises à jour correctives 9.4.4, 9.3.9, 9.2.13, 9.1.18 et 9.0.22 publiées. Déployez ! http://www.postgresql.org/docs/current/static/release.html

La classe d'administration de PostgreSQL de Consistent State aura lieu à Dallas (Texas, USA) du 21 au 24 septembre 2015 : http://www.consistentstate.com/store/p8/PostgreSQL_Admin_Class%3B_Dallas.html

Gianni Ciolli fera l'une des présentations introductives à la Dynamic Languages Conference du 20 juin à Manchester (Royaume-Uni) : http://www.dynamiclanguages.co.uk/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juin

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150614224208.GA22327@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Peter Eisentraut pushed:

Andrew Dunstan pushed:

  • Desupport jsonb subscript deletion on objects. Supporting deletion of JSON pairs within jsonb objects using an array-style integer subscript allowed for surprising outcomes. This was mostly due to the implementation-defined ordering of pairs within objects for jsonb. It also seems desirable to make jsonb integer subscript deletion consistent with the 9.4 era general purpose integer subscripting operator for jsonb (although that operator returns NULL when an object is encountered, while we prefer here to throw an error). Peter Geoghegan, following discussion on -hackers. http://git.postgresql.org/pg/commitdiff/b81c7b4098f52e64df89efe1461ba00a54649a10
  • Clarify documentation of jsonb - text. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/94d6727dbe61117addd9c24eea28440a2151ccf4
  • Rename jsonb - text[] operator to #- to avoid ambiguity. Following recent discussion on -hackers. The underlying function is also renamed to jsonb_delete_path. The regression tests now don't need ugly type casts to avoid the ambiguity, so they are also removed. Catalog version bumped. http://git.postgresql.org/pg/commitdiff/908e234733574545866045c7d5f93d4dd50ef66d
  • Fix "path" infrastructure bug affecting jsonb_set(). jsonb_set() and other clients of the setPathArray() utility function could get spurious results when an array integer subscript is provided that is not within the range of int. To fix, ensure that the value returned by strtol() within setPathArray() is within the range of int; when it isn't, assume an invalid input in line with existing, similar cases. The path-orientated operators that appeared in PostgreSQL 9.3 and 9.4 do not call setPathArray(), and already independently take this precaution, so no change there. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/2271d002d5c305441398e8f7a295f18ec3c132a9

Andres Freund pushed:

  • Allow HotStandbyActiveInReplay() to be called in single user mode. HotStandbyActiveInReplay, introduced in 061b079f, only allowed WAL replay to happen in the startup process, missing the single user case. This buglet is fairly harmless as it only causes problems when single user mode in an assertion enabled build is used to replay a btree vacuum record. Backpatch to 9.2. 061b079f was backpatched further, but the assertion was not. http://git.postgresql.org/pg/commitdiff/d1b958218ac183d0e88348341ff6ba31397086ad

Fujii Masao pushed:

  • Refactor WAL segment copying code. * Remove unused argument "dstfname" and related code from XLogFileCopy(). * Previously XLogFileCopy() returned a pstrdup'd string so that InstallXLogFileSegment() used it later. Since the pstrdup'd string was never free'd, there could be a risk of memory leak. It was almost harmless because the startup process exited just after calling XLogFileCopy(), it existed. This commit changes XLogFileCopy() so that it directly calls InstallXLogFileSegment() and doesn't call pstrdup() at all. Which fixes that memory leak problem. * Extend InstallXLogFileSegment() so that the caller can specify the log level. Which allows us to emit an error when InstallXLogFileSegment() fails a disk file access like link() and rename(). Previously it was always logged with LOG level and additionally needed to be logged with ERROR when we wanted to treat it as an error. Michael Paquier http://git.postgresql.org/pg/commitdiff/7abc68597436da1475b4d9b08f4fa9f3c5ed6185
  • Fix typo in comment. David Rowley http://git.postgresql.org/pg/commitdiff/ea9c4c1e4a7a9b602d867dcb02e07ef1fe51f6ec
  • Fix some issues in pg_rewind. * Remove invalid option character "N" from the third argument (valid option string) of getopt_long(). * Use pg_free() or pfree() to free the memory allocated by pg_malloc() or palloc() instead of always using free(). * Assume problem is no disk space if write() fails but doesn't set errno. * Fix several typos. Patch by me. Review by Michael Paquier. http://git.postgresql.org/pg/commitdiff/966c37fdb5ed9b87f3e91eace4dbbed7909f6769
  • Clean up useless mention of RMGRDESCSOURCES in pg_rewind Makefile. RMGRDESCSOURCES is defined and used only in pg_xlogdump Makefile, but pg_rewind Makefile mentioned it as extra files to remove in "make clean". This patch removes that useless mention from pg_rewind Makefile. Michael Paquier http://git.postgresql.org/pg/commitdiff/cd3cff4778e011c584e1481a6803dec5d4756a6e
  • Fix alphabetization in catalogs.sgml. System catalogs and views should be listed alphabetically in catalog.sgml, but only pg_file_settings view not. This patch also fixes typos in pg_file_settings comments. http://git.postgresql.org/pg/commitdiff/091c02a958fd0ae02b96492d9728efe8526385e6
  • Make postmaster restart archiver soon after it dies, even during recovery. After the archiver dies, postmaster tries to start a new one immediately. But previously this could happen only while server was running normally even though archiving was enabled always (i.e., archive_mode was set to always). So the archiver running during recovery could not restart soon after it died. This is an oversight in commit ffd3774. This commit changes reaper(), postmaster's signal handler to cleanup after a child process dies, so that it tries to a new archiver even during recovery if necessary. Patch by me. Review by Alvaro Herrera. http://git.postgresql.org/pg/commitdiff/b5fe62038f49f92c4a4f189c7cdacf3739effcdc

Ãlvaro Herrera pushed:

Tom Lane pushed:

  • First-draft release notes for 9.4.4, 9.3.9, 9.2.13, 9.1.18, 9.0.22. http://git.postgresql.org/pg/commitdiff/21187cfc7dfd82461db9119377a76366c00d27c3
  • Report more information if pg_perm_setlocale() fails at startup. We don't know why a few Windows users have seen this fail, but the taciturnity of the error message certainly isn't helping debug it. Let's at least find out which LC category isn't working. http://git.postgresql.org/pg/commitdiff/f6e9cbfd9153958226b27e31d658e7b64351c71f
  • Release notes for 9.4.4, 9.3.9, 9.2.13, 9.1.18, 9.0.22. http://git.postgresql.org/pg/commitdiff/b94085920b016e64ee40a0f6c50199889565cc56
  • Improve error message and hint for ALTER COLUMN TYPE can't-cast failure. We already tried to improve this once, but the "improved" text was rather off-target if you had provided a USING clause. Also, it seems helpful to provide the exact text of a suggested USING clause, so users can just copy-and-paste it when needed. Per complaint from Keith Rarick and a suggestion from Merlin Moncure. Back-patch to 9.2 where the current wording was adopted. http://git.postgresql.org/pg/commitdiff/b00982344a73d9cb626430dd17a6da84c15c9980
  • Fix failure to cover scalar-vs-rowtype cases in exec_stmt_return(). In commit 9e3ad1aac52454569393a947c06be0d301749362 I modified plpgsql to use exec_stmt_return's simple-variables fast path in more cases. However, I overlooked that there are really two different return conventions in use here, depending on whether estate->retistuple is true, and the existing fast-path code had only bothered to handle one of them. So trying to return a scalar in a function returning composite, or vice versa, could lead to unexpected error messages (typically "cache lookup failed for type 0") or to a null-pointer-dereference crash. In the DTYPE_VAR case, we can just throw error if retistuple is true, corresponding to what happens in the general-expression code path that was being used previously. (Perhaps someday both of these code paths should attempt a coercion, but today is not that day.) In the REC and ROW cases, just hand the problem to exec_eval_datum() when not retistuple. Also clean up the ROW coding slightly so it looks more like exec_eval_datum(). The previous commit also caused exec_stmt_return_next() to be used in more cases, but that code seems to be OK as-is. Per off-list report from Serge Rielau. This bug is new in 9.5 so no need to back-patch. http://git.postgresql.org/pg/commitdiff/ae58f1430abb4b0c309c40b377f91bf9d080334b

Bruce Momjian pushed:

Kevin Grittner pushed:

Michael Meskes pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Michael Paquier sent in two more revisions of a patch to fix some memory leaks in pg_dump, initdb and pg_upgrade.

Abhijit Menon-Sen sent in another revision of a patch to skip files in pg_log during basebackup.

Thomas Munro sent in two revisions of a patch to fix bogus truncation errors.

Dean Rasheed sent in a patch to fix an incompatibility between RLS and WHERE CURRENT OF.

Michael Paquier sent in a patch to restrict visibility of SSL information in pg_stat_ssl.

Andres Freund sent in a patch to improve multixact emergency autovacuum logic.

Jeevan Chalke sent in a patch to remove dead code in Create/RenameRole() after RoleSpec changes related to CURRENT/SESSION_USER.

Jeevan Chalke sent in another revision of a patch to add \ev and \sv (edit- and show-view) functionality to psql.

Michael Paquier sent in a patch to expand the use of xlog_internal.h's macros.

Alexander Shulgin sent in a patch to fix a problem where logical decoding sendtime update was always set to zero.

Gurjeet Singh sent in three revisions of a patch to initialize restart_lsn when the replication slot is created.

Peter Geoghegan sent in a patch to refactor speculative insertion with unique indexes.

Amit Kapila sent in another revision of a patch to remove only symlinks during recovery.

Etsuro Fujita sent in a patch to fix some comments in table inheritance.

Etsuro Fujita sent in a patch to fix a typo in comment in setrefs.c.

Jan Wieck sent in a patch to change some of the s_lock parameters with the aim of improving performance on modern NUMA machines.

Kaigai Kouhei sent in another revision of a patch to implement custom-join children.

Kaigai Kouhei sent in a patch to fix an issue where DBT-3 with SF=20 failed.

Naoya Anzai sent in a patch to make finding the backend PID a shortcut, \bid.

Andrew Dunstan sent in a patch to add json_count_array().

Peter Geoghegan sent in a patch to remove dead code for jsonb subscript deletion.

Peter Geoghegan sent in a patch to fully remove heap_formtuple() and friends.

Pavel Stehule sent in a patch to lower the log level for success in (un)registering background workers from LOG to DEBUG1.

Petr Jelínek sent in a patch to add tab completion for TABLESAMPLE to psql.

Dean Rasheed sent in a patch to add documentation for CREATE FUNCTION .. LEAKPROOF.

Ãlvaro Herrera sent in a patch to add a pg_get_multixact_range() function.

Petr Jelinek send in a flock of patches which: splits out pg_resetxlog into some reusable pieces and the calls to same, adds pg_resetsysid utility which changes the system id to newly generated one, adds -s option to pg_resetxlog to change the system id to the one specified, and adds fall-back implementation of the two functions based on very slightly modified code from OpenBSD.

Jeff Davis sent in another revision of a patch to implement memory accounting by tracking memory usage (at the block level) for all memory contexts.

par N Bougain le samedi 27 juin 2015 à 00h31

Nouvelles hebdomadaires de PostgreSQL - 7 juin 2015

Mises à jour correctives 9.4.3, 9.3.8, 9.2.12, 9.1.17 et 9.0.21 publiées. Déployez ! https://wiki.postgresql.org/wiki/May_2015_Fsync_Permissions_Bug

PGBR2015 (la PgConf brésilienne) aura lieu à Porto Alegre (État du Rio Grande do Sul) les 18, 19 et 20 novembre. L'appel à conférenciers expire le 15 juillet : http://pgbr.postgresql.org.br/2015/en/#call-for-papers

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juin

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150608005708.GA17798@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Andrew Dunstan pushed:

Bruce Momjian pushed:

Tom Lane pushed:

  • Release notes for 9.4.3, 9.3.8, 9.2.12, 9.1.17, 9.0.21. Also sneak entries for commits 97ff2a564 et al into the sections for the previous releases in the relevant branches. Those fixes did go out in the previous releases, but missed getting documented. http://git.postgresql.org/pg/commitdiff/82ec7d28211b97a2c9917b7a71edbe6b019578da
  • Fix planner's cost estimation for SEMI/ANTI joins with inner indexscans. When the inner side of a nestloop SEMI or ANTI join is an indexscan that uses all the join clauses as indexquals, it can be presumed that both matched and unmatched outer rows will be processed very quickly: for matched rows, we'll stop after fetching one row from the indexscan, while for unmatched rows we'll have an indexscan that finds no matching index entries, which should also be quick. The planner already knew about this, but it was nonetheless charging for at least one full run of the inner indexscan, as a consequence of concerns about the behavior of materialized inner scans --- but those concerns don't apply in the fast case. If the inner side has low cardinality (many matching rows) this could make an indexscan plan look far more expensive than it actually is. To fix, rearrange the work in initial_cost_nestloop/final_cost_nestloop so that we don't add the inner scan cost until we've inspected the indexquals, and then we can add either the full-run cost or just the first tuple's cost as appropriate. Experimentation with this fix uncovered another problem: add_path and friends were coded to disregard cheap startup cost when considering parameterized paths. That's usually okay (and desirable, because it thins the path herd faster); but in this fast case for SEMI/ANTI joins, it could result in throwing away the desired plain indexscan path in favor of a bitmap scan path before we ever get to the join costing logic. In the many-matching-rows cases of interest here, a bitmap scan will do a lot more work than required, so this is a problem. To fix, add a per-relation flag consider_param_startup that works like the existing consider_startup flag, but applies to parameterized paths, and set it for relations that are the inside of a SEMI or ANTI join. To make this patch reasonably safe to back-patch, care has been taken to avoid changing the planner's behavior except in the very narrow case of SEMI/ANTI joins with inner indexscans. There are places in compare_path_costs_fuzzily and add_path_precheck that are not terribly consistent with the new approach, but changing them will affect planner decisions at the margins in other cases, so we'll leave that for a HEAD-only fix. Back-patch to 9.3; before that, the consider_startup flag didn't exist, meaning that the second aspect of the patch would be too invasive. Per a complaint from Peter Holzer and analysis by Tomas Vondra. http://git.postgresql.org/pg/commitdiff/3f59be836c555fa679bbe0ec76de50a8b5cb23e0
  • Fix some questionable edge-case behaviors in add_path() and friends. add_path_precheck was doing exact comparisons of path costs, but it really needs to do them fuzzily to be sure it won't reject paths that could survive add_path's comparisons. (This can only matter if the initial cost estimate is very close to the final one, but that turns out to often be true.) Also, it should ignore startup cost for this purpose if and only if compare_path_costs_fuzzily would do so. The previous coding always ignored startup cost for parameterized paths, which is wrong as of commit 3f59be836c555fa6; it could result in improper early rejection of paths that we care about for SEMI/ANTI joins. It also always considered startup cost for unparameterized paths, which is just as wrong though the only effect is to waste planner cycles on paths that can't survive. Instead, it should consider startup cost only when directed to by the consider_startup/ consider_param_startup relation flags. Likewise, compare_path_costs_fuzzily should have symmetrical behavior for parameterized and unparameterized paths. In this case, the best answer seems to be that after establishing that total costs are fuzzily equal, we should compare startup costs whether or not the consider_xxx flags are on. That is what it's always done for unparameterized paths, so let's make the behavior for parameterized paths match. These issues were noted while developing the SEMI/ANTI join costing fix of commit 3f59be836c555fa6, but we chose not to back-patch these fixes, because they can cause changes in the planner's choices among nearly-same-cost plans. (There is in fact one minor change in plan choice within the core regression tests.) Destabilizing plan choices in back branches without very clear improvements is frowned on, so we'll just fix this in HEAD. http://git.postgresql.org/pg/commitdiff/3b0f77601b9f9f3a2e36a813e4cd32c00e0864d6
  • Stabilize query plans in rowsecurity regression test. Some recent buildfarm failures can be explained by supposing that autovacuum or autoanalyze fired on the tables created by this test, resulting in plan changes. Do a proactive VACUUM ANALYZE on the test's principal tables to try to forestall such changes. http://git.postgresql.org/pg/commitdiff/5cdf25e16843dff33dbc2ddc02941458032e3ad4
  • Stabilize results of brin regression test. This test used seqscans on tenk1, with LIMIT, to build test data. That works most of the time, but if the synchronized-seqscan logic kicks in, we get varying test data. This seems likely to explain the erratic test failures on buildfarm member chipmunk, which uses smaller-than-default shared_buffers. To fix, add ORDER BY clauses to force the ordering to be what it was implicitly being assumed to be. Peter Geoghegan had noticed this with respect to one of the trouble spots, though not the ones actually causing the chipmunk issue. http://git.postgresql.org/pg/commitdiff/bac99475eb6e9e6d69a91fee30b5420b8e0115be
  • Fix brin "char" test to actually test what it meant to test. Casting to char, without quotes, does not give the same results as casting to "char". That meant we were not testing the brin "char" paths at all, since we ended up with a text operator not a "char" operator. http://git.postgresql.org/pg/commitdiff/78e72794a76fef3233c06800c6046aaad0704a22
  • Tighten the per-operator testing done in brin regression test. Verify that the number of matches is exactly what it should be, not just that it not be zero. This should help us detect any environment-dependent issues. Also, verify that we're getting the expected type of scan plan (either bitmap or seqscan as appropriate). Right now, this is failing on the cidrcol test cases, as shown in the output file. I'll look into that in a bit, but it seems good to commit this as-is temporarily to verify that it behaves as expected on the buildfarm. http://git.postgresql.org/pg/commitdiff/79454c696bd3346a9f00f5e940398fb01a337fad
  • Fix brin regression test so it actually tests cidr. The problem noted in my previous commit was simpler than I thought: we weren't getting an index plan because the column wasn't indexed. http://git.postgresql.org/pg/commitdiff/1676e4381f48f7bf211f0965ad23abe10a683818
  • Second try at stabilizing query plans in rowsecurity regression test. This reverts commit 5cdf25e16843dff33dbc2ddc02941458032e3ad4, which was almost immediately proven insufficient by the buildfarm. On second thought, the tables involved are not large enough that autovacuum or autoanalyze would notice them; what seems far more likely to be the culprit is the database-wide "vacuum analyze" in the concurrent gist test. That thing has given us one headache too many, so get rid of it in favor of targeted vacuuming of that test's own tables only. http://git.postgresql.org/pg/commitdiff/1d27842519999cbac7e1cca8beaef053be9c7825
  • Fix incorrect order of database-locking operations in InitPostgres(). We should set MyProc->databaseId after acquiring the per-database lock, not beforehand. The old way risked deadlock against processes trying to copy or delete the target database, since they would first acquire the lock and then wait for processes with matching databaseId to exit; that left a window wherein an incoming process could set its databaseId and then block on the lock, while the other process had the lock and waited in vain for the incoming process to exit. CountOtherDBBackends() would time out and fail after 5 seconds, so this just resulted in an unexpected failure not a permanent lockup, but it's still annoying when it happens. A real-world example of a use-case is that short-duration connections to a template database should not cause CREATE DATABASE to fail. Doing it in the other order should be fine since the contract has always been that processes searching the ProcArray for a database ID must hold the relevant per-database lock while searching. Thus, this actually removes the former race condition that required an assumption that storing to MyProc->databaseId is atomic. It's been like this for a long time, so back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/ac23b711dd6ccb82fb70ca0f153fe755fd809a46
  • Get rid of a //-style comment. Not sure how "//XXX" got into a committed patch in the first place, as it's both content-free and against project style. pgindent made a bit of a hash of it, too. Going forward, we should have at least one buildfarm member using "gcc -ansi" to catch such things, at least till such time as we decide the project target language isn't C90 any more. I've turned this option on on dromedary. http://git.postgresql.org/pg/commitdiff/1497369e5df8bb129256f677a85327f80d3767d3
  • Use a safer method for determining whether relcache init file is stale. When we invalidate the relcache entry for a system catalog or index, we must also delete the relcache "init file" if the init file contains a copy of that rel's entry. The old way of doing this relied on a specially maintained list of the OIDs of relations present in the init file: we made the list either when reading the file in, or when writing the file out. The problem is that when writing the file out, we included only rels present in our local relcache, which might have already suffered some deletions due to relcache inval events. In such cases we correctly decided not to overwrite the real init file with incomplete data --- but we still used the incomplete initFileRelationIds list for the rest of the current session. This could result in wrong decisions about whether the session's own actions require deletion of the init file, potentially allowing an init file created by some other concurrent session to be left around even though it's been made stale. Since we don't support changing the schema of a system catalog at runtime, the only likely scenario in which this would cause a problem in the field involves a "vacuum full" on a catalog concurrently with other activity, and even then it's far from easy to provoke. Remarkably, this has been broken since 2002 (in commit 786340441706ac1957a031f11ad1c2e5b6e18314), but we had never seen a reproducible test case until recently. If it did happen in the field, the symptoms would probably involve unexpected "cache lookup failed" errors to begin with, then "could not open file" failures after the next checkpoint, as all accesses to the affected catalog stopped working. Recovery would require manually removing the stale "pg_internal.init" file. To fix, get rid of the initFileRelationIds list, and instead consult syscache.c's list of relations used in catalog caches to decide whether a relation is included in the init file. This should be a tad more efficient anyway, since we're replacing linear search of a list with ~100 entries with a binary search. It's a bit ugly that the init file contents are now so directly tied to the catalog caches, but in practice that won't make much difference. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/f3b5565dd4e59576be4c772da364704863e6a835

Fujii Masao pushed:

Peter Eisentraut pushed:

Robert Haas pushed:

  • docs: Fix list of object types pg_table_is_visible() can handle. Materialized views and foreign tables were missing from the list, probably because they are newer than the other object types that were mentioned. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/1c645da8ebb5532105481ad77bb1d9a671b1f086
  • doc: Session identifiers truncate, not round, the backend start time. Joel Jacobson http://git.postgresql.org/pg/commitdiff/99cfd5e136e2a20c77021390a1378d18a24b7388
  • Cope with possible failure of the oldest MultiXact to exist. Recent commits, mainly b69bf30b9bfacafc733a9ba77c9587cf54d06c0c and 53bb309d2d5a9432d2602c93ed18e58bd2924e15, introduced mechanisms to protect against wraparound of the MultiXact member space: the number of multixacts that can exist at one time is limited to 2^32, but the total number of members in those multixacts is also limited to 2^32, and older code did not take care to enforce the second limit, potentially allowing old data to be overwritten while it was still needed. Unfortunately, these new mechanisms failed to account for the fact that the code paths in which they run might be executed during recovery or while the cluster was in an inconsistent state. Also, they failed to account for the fact that users who used pg_upgrade to upgrade a PostgreSQL version between 9.3.0 and 9.3.4 might have might oldestMultiXid = 1 in the control file despite the true value being larger. To fix these problems, first, avoid unnecessarily examining the mmembers of MultiXacts when the cluster is not known to be consistent. TruncateMultiXact has done this for a long time, and this patch does not fix that. But the new calls used to prevent member wraparound are not needed until we reach normal running, so avoid calling them earlier. (SetMultiXactIdLimit is actually called before InRecovery is set, so we can't rely on that; we invent our own multixact-specific flag instead.) Second, make failure to look up the members of a MultiXact a non-fatal error. Instead, if we're unable to determine the member offset at which wraparound would occur, postpone arming the member wraparound defenses until we are able to do so. If we're unable to determine the member offset that should force autovacuum, force it continuously until we are able to do so. If we're unable to deterine the member offset at which we should truncate the members SLRU, log a message and skip truncation. An important consequence of these changes is that anyone who does have a bogus oldestMultiXid = 1 value in pg_control will experience immediate emergency autovacuuming when upgrading to a release that contains this fix. The release notes should highlight this fact. If a user has no pg_multixact/offsets/0000 file, but has oldestMultiXid = 1 in the control file, they may wish to vacuum any tables with relminmxid = 1 prior to upgrading in order to avoid an immediate emergency autovacuum after the upgrade. This must be done with a PostgreSQL version 9.3.5 or newer and with vacuum_multixact_freeze_min_age and vacuum_multixact_freeze_table_age set to 0. This patch also adds an additional log message at each database server startup, indicating either that protections against member wraparound have been engaged, or that they have not. In the latter case, once autovacuum has advanced oldestMultiXid to a sane value, the message indicating that the guards have been engaged will appear at the next checkpoint. A few additional messages have also been added at the DEBUG1 level so that the correct operation of this code can be properly audited. Along the way, this patch fixes another, related bug in TruncateMultiXact that has existed since PostgreSQL 9.3.0: when no MultiXacts exist at all, the truncation code looks up NextMultiXactId, which doesn't exist yet. This can lead to TruncateMultiXact removing every file in pg_multixact/offsets instead of keeping one around, as it should. This in turn will cause the database server to refuse to start afterwards. Patch by me. Review by Ãlvaro Herrera, Andres Freund, Noah Misch, and Thomas Munro. http://git.postgresql.org/pg/commitdiff/068cfadf9e2190bdd50a30d19efc7c9f0b825b5e

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Peter Geoghegan sent in a patch to illustrate a possible security bug in RLS.

Dean Rasheed sent in a patch to refactor the implementation of RLS.

Laurenz Albe sent in a patch to allow using literal tabs in psql.

Etsuro Fujita sent in a doc patch to add materialized views and foreign tables to database objects that pg_table_is_visible() can be used with.

Michael Paquier sent in two more revisions of a patch to fix a memory leak in XLogFileCopy.

Fabien COELHO sent in two revisions of a patch to enable flushing while writing during checkpoints.

Andreas Seltenreich sent in a patch to add error handling to byteaout.

Michael Paquier sent in a patch to remove the use of %.*s in several parts of the psql code to make the code more solid when facing non-ASCII strings.

Craig Ringer sent in another revision of a patch to allow sampling of only some queries by auto_explain.

Peter Geoghegan sent in a patch to desupport jsonb subscript deletion on objects, which was causing surprising outcomes.

Jeevan Chalke sent in another revision of a patch to implement a two-argument version of current_setting() with fallback.

Kaigai Kouhei sent in another revision of a patch to allow custom-join children.

Amit Kapila and Andrew Dunstan traded patches to remove only symlinks during recovery.

Julien Rouhaud sent in a patch fix an issue where when archiver aborts, pg_stat_archiver doesn't report those failed attempts.

Petr Korobeinikov sent in another revision of a patch to implement \ev and \sv (edit view, and show view, respectively) in psql.

Peter Geoghegan sent in a patch to add a regression test in cases where RLS again fails to play nicely with UPDATE ... WHERE CURRENT OF.

Thomas Munro sent in a patch to fix a bogus subtrans wraparound error.

par N Bougain le samedi 27 juin 2015 à 00h04

vendredi 26 juin 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 31 mai 2015

Mises à jour correctives 9.4.3, 9.3.8, 9.2.12, 9.1.17 et 9.0.21 bientôt disponibles. Préparez le déploiement : https://wiki.postgresql.org/wiki/May_2015_Fsync_Permissions_Bug

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150531225635.GB8579@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Bruce Momjian pushed:

Heikki Linnakangas pushed:

Andrew Dunstan pushed:

Ãlvaro Herrera pushed:

  • Update README.tuplock. Multixact truncation is now handled differently, and this file hadn't gotten the memo. Per note from Amit Langote. I didn't use his patch, though. Also update the description of infomask bits, which weren't completely up to date either. This commit also propagates b01a4f6838 back to 9.3 and 9.4, which apparently I failed to do back then. http://git.postgresql.org/pg/commitdiff/cdbdc4382743fcfabb3437ea7c4d103adaa01324

Tom Lane pushed:

  • Explain CHECK constraint handling in postgres_fdw's IMPORT FOREIGN SCHEMA. The existing documentation could easily be misinterpreted, and it failed to explain the inconsistent-evaluation hazard that deterred us from supporting automatic importing of check constraints. Revise it. Etsuro Fujita, further expanded by me http://git.postgresql.org/pg/commitdiff/e70ec8230a2c0e7363bb7abf4d55dddbdec89fe1
  • Fix valgrind's "unaddressable bytes" whining about BRIN code. brin_form_tuple calculated an exact tuple size, then palloc'd and filled just that much. Later, brin_doinsert or brin_doupdate would MAXALIGN the tuple size and tell PageAddItem that that was the size of the tuple to insert. If the original tuple size wasn't a multiple of MAXALIGN, the net result would be that PageAddItem would memcpy a few more bytes than the palloc request had been for. AFAICS, this is totally harmless in the real world: the error is a read overrun not a write overrun, and palloc would certainly have rounded the request up to a MAXALIGN multiple internally, so there's no chance of the memcpy fetching off the end of memory. Valgrind, however, is picky to the byte level not the MAXALIGN level. Fix it by pushing the MAXALIGN step back to brin_form_tuple. (The other possible source of tuples in this code, brin_form_placeholder_tuple, was already producing a MAXALIGN'd result.) In passing, be a bit more paranoid about internal allocations in brin_form_tuple. http://git.postgresql.org/pg/commitdiff/79f2b5d583e2e2a7ccd13e31d0e20a900c8f2f61
  • Suppress occasional failures in brin regression test. brin.sql included a call of brin_summarize_new_values(), and expected it to always report exactly 5 summarization events. This failed sometimes during parallel regression tests, as a consequence of the database-wide VACUUM in gist.sql getting there first. The most future-proof way to avoid variation in the test results is to forget about using brin_summarize_new_values() and just do a plain "VACUUM brintest", which will exercise the same code anyway. Having done that, there's no need for preventing autovacuum on brintest; doing so just reduces the scope of test coverage, so let's not. http://git.postgresql.org/pg/commitdiff/1f303fd1be51f26553e7c95d8696aa4e28ece1c6
  • Remove configure check prohibiting threaded libpython on OpenBSD. According to recent tests, this case now works fine, so there's no reason to reject it anymore. (Even if there are still some OpenBSD platforms in the wild where it doesn't work, removing the check won't break any case that worked before.) We can actually remove the entire test that discovers whether libpython is threaded, since without the OpenBSD case there's no need to know that at all. Per report from Davin Potts. Back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/86832eb8912b9cac0f4961facb9efda81828e214
  • Fix portability issue in isolationtester grammar. specparse.y and specscanner.l used "string" as a token name. Now, bison likes to define each token name as a macro for the token code it assigns, which means those names are basically off-limits for any other use within the grammar file or included headers. So names as generic as "string" are dangerous. This is what was causing the recent failures on protosciurus: some versions of Solaris' sys/kstat.h use "string" as a field name. With late-model bison we don't see this problem because the token macros aren't defined till later (that is why castoroides didn't show the problem even though it's on the same machine). But protosciurus uses bison 1.875 which defines the token macros up front. This land mine has been there from day one; we'd have found it sooner except that protosciurus wasn't trying to run the isolation tests till recently. To fix, rename the token to "string_literal" which is hopefully less likely to collide with names used by system headers. Back-patch to all branches containing the isolation tests. http://git.postgresql.org/pg/commitdiff/aa9eac45ea868e6ddabc4eb076d18be10ce84c6a
  • Fix pg_get_functiondef() to print a function's LEAKPROOF property. Seems to have been an oversight in the original leakproofness patch. Per report and patch from Jeevan Chalke. In passing, prettify some awkward leakproof-related code in AlterFunction. http://git.postgresql.org/pg/commitdiff/f46edf479e2468a08caca2a03ec7e258930a7161
  • Fix assorted inconsistencies in our calls of readlink(). Ensure that we null-terminate the result string (one place in pg_rewind). Be paranoid about out-of-range results from readlink() (should not happen, but there is no good reason for some call sites to be careful about it and others not). Consistently use the whole buffer, not sometimes one byte less. Ensure we emit an appropriate errcode() in all cases. Spell the error messages the same way. The only serious bug here is the missing null-termination in pg_rewind, which is new code, so no need for a back-patch. Abhijit Menon-Sen and Tom Lane http://git.postgresql.org/pg/commitdiff/32f628be74d8ab43423ca7913b81f7eb21b312d4
  • Fix pg_rewind's handling of top-level symlinks. The previous coding suffered a null-pointer dereference if it found any symlink at the top level of $PGDATA. Fix that, and teach it to recurse into a symlink for pg_xlog, but not anything else. Per note from Abhijit Menon-Sen. http://git.postgresql.org/pg/commitdiff/0381fefaa44f04e17dffb7e46e7677374a4fb2c7
  • Fix fsync-at-startup code to not treat errors as fatal. Commit 2ce439f3379aed857517c8ce207485655000fc8e introduced a rather serious regression, namely that if its scan of the data directory came across any un-fsync-able files, it would fail and thereby prevent database startup. Worse yet, symlinks to such files also caused the problem, which meant that crash restart was guaranteed to fail on certain common installations such as older Debian. After discussion, we agreed that (1) failure to start is worse than any consequence of not fsync'ing is likely to be, therefore treat all errors in this code as nonfatal; (2) we should not chase symlinks other than those that are expected to exist, namely pg_xlog/ and tablespace links under pg_tblspc/. The latter restriction avoids possibly fsync'ing a much larger part of the filesystem than intended, if the user has left random symlinks hanging about in the data directory. This commit takes care of that and also does some code beautification, mainly moving the relevant code into fd.c, which seems a much better place for it than xlog.c, and making sure that the conditional compilation for the pre_sync_fname pass has something to do with whether pg_flush_data works. I also relocated the call site in xlog.c down a few lines; it seems a bit silly to be doing this before ValidateXLOGDirectoryStructure(). The similar logic in initdb.c ought to be made to match this, but that change is noncritical and will be dealt with separately. Back-patch to all active branches, like the prior commit. Abhijit Menon-Sen and Tom Lane http://git.postgresql.org/pg/commitdiff/d8179b001ae574da00c8f4549577095bf90f3337
  • Revert exporting of internal GUC variable "data_directory". This undoes a poorly-thought-out choice in commit 970a18687f9b3058, namely to export guc.c's internal variable data_directory. The authoritative variable so far as C code is concerned is DataDir; there is no reason for anything except specific bits of GUC code to look at the GUC variable. After yesterday's commits fixing the fsync-on-restart patch, the only remaining misuse of data_directory was in AlterSystemSetConfigFile(), which would be much better off just using a relative path anyhow: it's less code and it doesn't break if the DBA moves the data directory of a running system, which is a case we've taken some pains over in the past. This is mostly cosmetic, so no need for a back-patch (and I'd be hesitant to remove a global variable in stable branches anyway). http://git.postgresql.org/pg/commitdiff/da33a3894e0fc440ac53cdc0f2e360e703b13b8c
  • Check that all aliases of a built-in function have same leakproof property. opr_sanity.sql has a test checking that relevant properties of built-in functions match when the same C function is referenced by multiple pg_proc entries. The test neglected to check proleakproof, though, and when I added that condition it exposed that xideqint4 hadn't been updated to match xideq. So fix that as well, and in consequence bump catversion. This isn't very critical, so no need to worry about fixing back branches. http://git.postgresql.org/pg/commitdiff/1c8c656b3c217aaffc503ad703dcc60cdabe7445
  • Remove special cases for ETXTBSY from new fsync'ing logic. The argument that this is a sufficiently-expected case to be silently ignored seems pretty thin. Andres had brought it up back when we were still considering that most fsync failures should be hard errors, and it probably would be legit not to fail hard for ETXTBSY --- but the same is true for EROFS and other cases, which is why we gave up on hard failures. ETXTBSY is surely not a normal case, so logging the failure seems fine from here. http://git.postgresql.org/pg/commitdiff/57e1138bcc621ffeb8b1f1379ac4016a5c34d43e
  • initdb -S should now have an explicit check that $PGDATA is valid. The fsync code from the backend essentially assumes that somebody's already validated PGDATA, at least to the extent of it being a readable directory. That's safe enough for initdb's normal code path too, but "initdb -S" doesn't have any other processing at all that touches the target directory. To have reasonable error-case behavior, add a pg_check_dir call. Per gripe from Peter E. http://git.postgresql.org/pg/commitdiff/1943c000b7a22d3ca334196cfe3f7b8159b210c2
  • Adjust initdb to also not consider fsync'ing failures fatal. Make initdb's version of this logic look as much like the backend's as possible. This is much less critical than in the backend since not so many people use "initdb -S", but we want the same corner-case error handling in both cases. Back-patch to 9.3 where initdb -S option was introduced. Before that, initdb only had to deal with freshly-created data directories, wherein no failures should be expected. Abhijit Menon-Sen http://git.postgresql.org/pg/commitdiff/c07d8c963e39980f192e8daca73b7585ef76cc9b

Stephen Frost pushed:

Peter Eisentraut pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Peter Geoghegan sent in a patch to point out that for foreign tables, AddForeignUpdateTargets is not called INSERT operations with an ON CONFLICT DO UPDATE clause because that clause is not yet supported on foreign tables.

Peter Geoghegan sent in a flock of patches intended to ensure that some of the ON CONFLICT UPDATE always has its memory freed by its caller.

Emre Hasegeli sent in a patch to ensure that adjacent strategy numbers of range types match the central definitions.

Tomas Vondra sent in another revision of a patch to implement multivariate statistics.

Michael Paquier sent in another revision of a patch to add support for TAP tests on Windows.

Kaigai Kouhei sent in another revision of a patch to add a custom child jointree.

Uriy Zhuravlev sent in another revision of a patch to implement ALTER OPERATOR.

Uriy Zhuravlev sent in a patch to to add selectivity functions for intarray.

Andrew Dunstan sent in a patch to add a jsonb_set() function.

Tatsuo Ishii sent in a patch to fix some missing parts of the psql po translation for the Japanese language.

Peter Eisentraut sent in a WIP candidate candidate configuration which might eventually replace pgindent with clang-format.

SAWADA Masahiko sent in another revision of a patch to add a "frozen" bit into the visibility map.

Shigeru HANADA sent in another revision of a patch to implement postgres_fdw join pushdown.

boix sent in two revisions of a patch to use an ant colony optimization for GEQO.

Michael Paquier sent in another revision of a doc patch to better describe the behavior of txid_current().

Michael Paquier sent in a patch to fix a memory leak in XLogFileCopy.

Craig Ringer sent in a patch to document that directly callable functions may use fn_extra.

Naoya Anzai sent in a patch to add more vacuum statistics.

Abhijit Menon-Sen sent in a patch to ensure that S_IRUSR|S_IWUSR are set only with O_CREAT.

Peter Eisentraut sent in a patch to allow ALTER TABLE ENABLE TRIGGER to work on VIEWs.

Thomas Munro and Robert Haas traded patches to tolerate missing offset segments.

Tomas Vondra and Tom Lane traded patches to fix cost estimates for nested loop semijoin.

Andreas Karlsson sent in a patch to reload SSL certificates on SIGHUP, which makes it possible to enable or disable SSL entirely without a restart.

Andreas Karlsson sent in a patch to fix autoconf deprecation warnings.

par N Bougain le vendredi 26 juin 2015 à 23h35

mardi 16 juin 2015

Guillaume Lelarge

Différences entre les versions beta du livre

Je me suis rendu compte ce week-end qu'on n'avait pas publié d'informations sur ce qui avait été ajouté entre les différentes versions beta, en dehors des nouveaux chapitres. Voici donc la liste des modifications, un peu éditée pour être plus lisible :

Pour la beta2 :

  • Nouveaux chapitres
    • protocole de communication
    • connexions
  • Chapitre fichiers
    • ajout d'une note sur l'option --no-clean de la commande initdb
    • ajout d'une note sur les versions 9.1 (et inférieures) et la colonne spclocation du catalogue pg_tablespace
    • ajout d'une note sur le fichier pgstat.stat des versions 9.3 et antérieures
  • Chapitre processus
    • refonte des sections pour trier les processus par activité (et non par nom)
    • revue du résumé sur les processus d'écriture dans les fichiers de données suite à une remarque d'un lecteur dans le forum du livre (forum uniquement accessible par les lecteurs actuels)
    • correction des processus équivalents au niveau Oracle, suite là-aussi à un autre commentaire d'un lecteur
  • Chapitre mémoire
    • ajout de deux paragraphes sur l'utilisation (partielle) du cache pour les requêtes
    • présentation de deux colonnes de la vue pg_stat_database permettant de quantifier l'utilisation des fichiers temporaires

Et pour la beta 3 :

  • Nouveaux chapitres
    • gestion des objets
    • transactions
  • Global
    • remplacement du terme maître par serveur primaire et du terme esclave par serveur secondaire (suite à la demande d'un lecteur)
  • Chapitre mémoire
    • ajout d'une note sur l'intérêt du paramètre maintenance_work_mem dans le cadre d'un import de données avec pg_restore
  • Chapitre processus
    • ajout d'une partie sur les écritures dans les fichiers de données suite à un CHECKPOINT, avec quelques graphes, pour mieux expliquer les différents paramètres checkpoint_*
  • Chapitre Fichiers
    • ajout d'un paragraphe sur la génération des relfilnode
    • ajout d'infos sur le LSN et les journaux de transactions

Je publierais dans ce blog les nouveautés de chaque version beta à chaque sortie, ce sera plus facile pour les lecteurs.

par Guillaume Lelarge le mardi 16 juin 2015 à 20h32

vendredi 12 juin 2015

Actualités PostgreSQL.fr

Nouvelles versions mineures au 12 juin 2015

Le PostgreSQL Global Development Group a publié une mise à jour pour toutes les versions supportées du SGBD, soit les versions 9.4.4, 9.3.9, 9.2.13, 9.1.18 et 9.0.22.
Cette mise à jour a pour but principal de corriger les bogues qui n'ont pas été corrigés par les mises à jour précédentes.

Il est impératif de les appliquer immédiatement pour les utilisateurs ayant déjà appliqué les mises à jour de mai et juin.
Les autres utilisateurs pourront attendre le prochain arrêt de service programmé.

Correctifs des récupérations en cas de panne

Les mises à jour précédentes ont tenté de corriger un problème de "multixact wraparound" (ou bouclage d'identifiant de transactions multiples) sur les versions 9.3 et 9.4 de PostgreSQL.
Mais elles n'ont pas corrigé le nettoyage des transactions multiples lors de récupération apès crash. Cela peut empêcher un serveur de redémarrer après un crash.
C'est pourquoi il est recommandé aux utilisateurs de ces versions de procéder à la mise à jour le plus vite possible.

Les serveurs précédemment passés en version 9.3 avec pg_upgrade, ou ceux qui tournent en 9.4, suite à une autre mise à jour, peuvent voir tourner un autovacuum sur l'ensemble des tables immédiatement après l'application de l'actuelle mise à jour.
Si les bases sont volumineuses, il peut être plus judicieux de procéder à un VACUUM manuel, avant la mise à jour, pour maîtriser les performances suite à cette maintenance.

N'hésitez pas à consulter les notes de versions pour les détails ( Notes de version ).

Autres correctifs et améliorations

En plus des correctifs précédents, quelques autres correctifs ont été appliqués à cette version.

On trouve :

 * le correctif des rares cas ou l'invalidation du cache des relations n'était pas effectué ;
 * la suppression des verrous morts entre de nouvelles sessions et un CREATE/DROP DATABASE ;
 * l'amélioration de la planification des requêtes pour les semi-jointures et anti-jointures.

Mises à jour cumulatives

Toutes les mises à jour de PostgreSQL sont cumulatives. Comme cette mise à jour fixe un certain nombre de problèmes malencontreusement apportées par les mises à jour précédentes, il est préférable d'installer directement cette mise à jour plutôt que les précédentes.

Comme cette mise à jour clôt plusieurs bogues de gestion des transactions multiples, le projet PostgreSQL ne prévoit pas de nouvelles mises à jour prochainement.

Application de la mise à jour

Comme pour les autres mises à jour, il n'est pas nécessaire de procéder à l'export et au réimport des données, ou d'utiliser pg_upgrade. Il suffit d'installer les nouveaux binaires et de redémarrer PostgreSQL.

Au cas où des mises à jour mineures auraient été omises, il peut y avoir des opérations suppplémentaires à effectuer. Toutes les informations se trouvent dans les notes de versions. Voir également les informations complémentaires en cas de passage à la version 9.3 avec pg_upgrade.

Liens :

 * Téléchargement
 * Notes de version

par SAS le vendredi 12 juin 2015 à 14h35

vendredi 29 mai 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 24 mai 2015

Les mises à jour de sécurité et de correction 9.4.2, 9.3.7, 9.2.11, 9.1.16 et 9.0.20 sont disponibles. Déployez au plus tôt : http://www.postgresql.org/about/news/1587/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Use += not = to set makefile variables after including base makefiles. The previous coding in hstore_plpython and ltree_plpython wiped out any values set by the base makefiles. This at least had the effect of running the tests in "regression" not "contrib_regression" as expected. These being pretty new modules, there might be other bad effects we'd not noticed yet. http://git.postgresql.org/pg/commitdiff/b14cf229f4bd7238be2e31d873dc5dd241d3871e
  • Fix failure to copy IndexScan.indexorderbyops in copyfuncs.c. This oversight results in a crash at executor startup if the plan has been copied. outfuncs.c was missed as well. While we could probably have taught both those files to cope with the originally chosen representation of an Oid array, it would have been painful, not least because there'd be no easy way to verify the array length. An Oid List is far easier to work with. And AFAICS, there is no particular notational benefit to using an array rather than a list in the existing parts of the patch either. So just change it to a list. Error in commit 35fcb1b3d038a501f3f4c87c05630095abaaadab, which is new, so no need for back-patch. http://git.postgresql.org/pg/commitdiff/424661913c06af76a46fdff9cc24cc57abf14fb3
  • Recognize "REGRESS_OPTS += ..." syntax in MSVC build scripts. Necessitated by commit b14cf229f4bd7238be2e31d873dc5dd241d3871e. Per buildfarm. http://git.postgresql.org/pg/commitdiff/f5916bb7b53f8a77c95c00c5b287659958891178
  • Put back a backwards-compatible version of sampling support functions. Commit 83e176ec18d2a91dbea1d0d1bd94c38dc47cd77c removed the longstanding support functions for block sampling without any consideration of the impact this would have on third-party FDWs. The new API is not notably more functional for FDWs than the old, so forcing them to change doesn't seem like a good thing. We can provide the old API as a wrapper (more or less) around the new one for a minimal amount of extra code. http://git.postgresql.org/pg/commitdiff/4db485e75b9672126963ae4052b50f473b30a097
  • Revert "Change pg_seclabel.provider and pg_shseclabel.provider to type "name"." This reverts commit b82a7be603f1811a0a707b53c62de6d5d9431740. There is a better (less invasive) way to fix it, which I will commit next. http://git.postgresql.org/pg/commitdiff/afee04352bc01b79cde33c018a82c2eeb1ce84eb
  • Avoid collation dependence in indexes of system catalogs. No index in template0 should have collation-dependent ordering, especially not indexes on shared catalogs. For most textual columns we avoid this issue by using type "name" (which sorts per strcmp()). However there are a few indexed columns that we'd prefer to use "text" for, and for that, the default opclass text_ops is unsafe. Fortunately, text_pattern_ops is safe (it sorts per memcmp()), and it has no real functional disadvantage for our purposes. So change the indexes on pg_seclabel.provider and pg_shseclabel.provider to use text_pattern_ops. In passing, also mark pg_replication_origin.roname as using text_pattern_ops --- for some reason it was labeled varchar_pattern_ops which is just wrong, even though it accidentally worked. Add regression test queries to catch future errors of these kinds. We still can't do anything about the misdeclared pg_seclabel and pg_shseclabel indexes in back branches
  • http://git.postgresql.org/pg/commitdiff/0b28ea79c044a0d3779081dc909a6dc0ce93b991
  • - Change pg_seclabel.provider and pg_shseclabel.provider to type "name". These were "text", but that's a bad idea because it has collation-dependent ordering. No index in template0 should have collation-dependent ordering, especially not indexes on shared catalogs. There was general agreement that provider names don't need to be longer than other identifiers, so we can fix this at a small waste of table space by changing from text to name. There's no way to fix the problem in the back branches, but we can hope that security labels don't yet have widespread-enough usage to make it urgent to fix. There needs to be a regression sanity test to prevent us from making this same mistake again; but before putting that in, we'll need to get rid of similar brain fade in the recently-added pg_replication_origin catalog. Note: for lack of a suitable testing environment, I've not really exercised this change. I trust the buildfarm will show up any mistakes. http://git.postgresql.org/pg/commitdiff/b82a7be603f1811a0a707b53c62de6d5d9431740
  • Another typo fix. In the spirit of the season. http://git.postgresql.org/pg/commitdiff/a6a66bd647d471aeb55d8ba3e24d197ccd8a5abb
  • Improve packing/alignment annotation for ItemPointerData. We want this struct to be exactly a series of 3 int16 words, no more and no less. Historically, at least, some ARM compilers preferred to pad it to 8 bytes unless coerced. Our old way of doing that was just to use __attribute__((packed)), but as pointed out by Piotr Stefaniak, that does too much: it also licenses the compiler to give the struct only byte-alignment. We don't want that because it adds access overhead, possibly quite significant overhead. According to the GCC manual, what we want requires also specifying __attribute__((align(2))). It's not entirely clear if all the relevant compilers accept this pragma as well, but we can hope the buildfarm will tell us if not. We can also add a static assertion that should fire if the compiler padded the struct. Since the combination of these pragmas should define exactly what we want on any compiler that accepts them, let's try using them wherever we think they exist, not only for __arm__. (This is likely to expose that the conditional definitions in c.h are inadequate, but finding that out would be a good thing.) The immediate motivation for this is that the current definition of ExecRowMark allows its curCtid field to be misaligned. It is not clear whether there are any other uses of ItemPointerData with a similar hazard. We could change the definition of ExecRowMark if this doesn't work, but it would be far better to have a future-proof fix. Piotr Stefaniak, some further hacking by me http://git.postgresql.org/pg/commitdiff/d4b538ea367de43b2f2b939621272682417cd290
  • More fixes for lossy-GiST-distance-functions patch. Paul Ramsey reported that commit 35fcb1b3d038a501f3f4c87c05630095abaaadab induced a core dump on commuted ORDER BY expressions, because it was assuming that the indexorderby expression could be found verbatim in the relevant equivalence class, but it wasn't there. We really don't need anything that complicated anyway; for the data types likely to be used for index ORDER BY operators in the foreseeable future, the exprType() of the ORDER BY expression will serve fine. (The case where we'd have to work harder is where the ORDER BY expression's result is only binary-compatible with the declared input type of the ordering operator; long before worrying about that, one would need to get rid of GiST's hard-wired assumption that said datatype is float8.) Aside from fixing that crash and adding a regression test for the case, I did some desultory code review: nodeIndexscan.c was likewise overthinking how hard it ought to work to identify the datatype of the ORDER BY expressions. Add comments explaining how come nodeIndexscan.c can get away with simplifying assumptions about NULLS LAST ordering and no backward scan. Revert no-longer-needed changes of find_ec_member_for_tle(); while the new definition was no worse than the old, it wasn't better either, and it might cause back-patching pain. Revert entirely bogus additions to genam.h. http://git.postgresql.org/pg/commitdiff/c5dd8ead403f85bd041590d2e3e79b72830472d4
  • Fix recently-introduced crash in array_contain_compare(). Silly oversight in commit 1dc5ebc9077ab742079ce5dac9a6664248d42916: when array2 is an expanded array, it might have array2->xpn.dnulls equal to NULL, indicating the array is known null-free. The code wasn't expecting that, because it formerly always used deconstruct_array() which always delivers a nulls array. Per bug #13334 from Regina Obe. http://git.postgresql.org/pg/commitdiff/49ad32d5d99cb4a79bf648c0b7f9eca19b54cf1d
  • Last-minute updates for release notes. Add entries for security issues. Security: CVE-2015-3165 through CVE-2015-3167 http://git.postgresql.org/pg/commitdiff/19d47ed2da1e4d08ffab7e8ba1b1c4c614e7f296
  • Last-minute updates for release notes. Revise description of CVE-2015-3166, in line with scaled-back patch. Change release date. Security: CVE-2015-3166 http://git.postgresql.org/pg/commitdiff/5cb8519ceb62516636362a7e8e06b99b3e1bf138
  • Revert error-throwing wrappers for the printf family of functions. This reverts commit 16304a013432931e61e623c8d85e9fe24709d9ba, except for its changes in src/port/snprintf.c; as well as commit cac18a76bb6b08f1ecc2a85e46c9d2ab82dd9d23 which is no longer needed. Fujii Masao reported that the previous commit caused failures in psql on OS X, since if one exits the pager program early while viewing a query result, psql sees an EPIPE error from fprintf --- and the wrapper function thought that was reason to panic. (It's a bit surprising that the same does not happen on Linux.) Further discussion among the security list concluded that the risk of other such failures was far too great, and that the one-size-fits-all approach to error handling embodied in the previous patch is unlikely to be workable. This leaves us again exposed to the possibility of the type of failure envisioned in CVE-2015-3166. However, that failure mode is strictly hypothetical at this point: there is no concrete reason to believe that an attacker could trigger information disclosure through the supposed mechanism. In the first place, the attack surface is fairly limited, since so much of what the backend does with format strings goes through stringinfo.c or psprintf(), and those already had adequate defenses. In the second place, even granting that an unprivileged attacker could control the occurrence of ENOMEM with some precision, it's a stretch to believe that he could induce it just where the target buffer contains some valuable information. So we concluded that the risk of non-hypothetical problems induced by the patch greatly outweighs the security risks. We will therefore revert, and instead undertake closer analysis to identify specific calls that may need hardening, rather than attempt a universal solution. We have kept the portion of the previous patch that improved snprintf.c's handling of errors when it calls the platform's sprintf(). That seems to be an unalloyed improvement. Security: CVE-2015-3166 http://git.postgresql.org/pg/commitdiff/0c071936e94c6859afb2ec8d2c8dddf7bcdab7ee
  • Still more fixes for lossy-GiST-distance-functions patch. Fix confusion in documentation, substantial memory leakage if float8 or float4 are pass-by-reference, and assorted comments that were obsoleted by commit 98edd617f3b62a02cb2df9b418fcc4ece45c7ec0. http://git.postgresql.org/pg/commitdiff/821b821a2421beaa58225ff000833df69fb962c5
  • Fix incorrect snprintf() limit. Typo in commit 7cbee7c0a. No practical effect since the buffer should never actually be overrun, but various compilers and static analyzers will whine about it. Petr Jelinek http://git.postgresql.org/pg/commitdiff/72809480d658fbc0654239b2f089991c077c676a
  • Add error check for lossy distance functions in index-only scans. Maybe we should actually support this, but for the moment let's just throw an error if the opclass tries it. http://git.postgresql.org/pg/commitdiff/f84c8601d604811a530dadb53ddb52f08639e72b
  • Remove no-longer-required function declarations. Remove a bunch of "extern Datum foo(PG_FUNCTION_ARGS);" declarations that are no longer needed now that PG_FUNCTION_INFO_V1(foo) provides that. Some of these were evidently missed in commit e7128e8dbb305059, but others were cargo-culted in in code added since then. Possibly that can be blamed in part on the fact that we'd not fixed relevant documentation examples, which I've now done. http://git.postgresql.org/pg/commitdiff/91e79260f636ab4d5a43910b6a38bc75651ad14c
  • Add a bit more commentary about regex's colormap tree data structure. Per an off-list question from Piotr Stefaniak. http://git.postgresql.org/pg/commitdiff/23116d5437d0e8d077e7fd5391f5fa0fc781b7d2
  • Rename pg_shdepend.c's typedef "objectType" to SharedDependencyObjectType. The name objectType is widely used as a field name, and it's pure luck that this conflict has not caused pgindent to go crazy before. It messed up pg_audit.c pretty good though. Since pg_shdepend.c doesn't export this typedef and only uses it in three places, changing that seems saner than changing the field usages. Back-patch because we're contemplating using the union of all branch typedefs for future pgindent runs, so this won't fix anything if it stays the same in back branches. http://git.postgresql.org/pg/commitdiff/17b48a1a9f87f7479d38dcc78a27c23f1f8124f8
  • Manual cleanup of pgindent results. Fix some places where pgindent did silly stuff, often because project style wasn't followed to begin with. (I've not touched the atomics headers, though.) http://git.postgresql.org/pg/commitdiff/2aa0476dc38f7e510b8cde627e83b4c76fa05d61

Peter Eisentraut a poussé :

Fujii Masao a poussé :

Heikki Linnakangas a poussé :

  • Put back stats-collector restarting code, removed accidentally. Removed that code snippet accidentally in the archive_mode='always' patch. Also, use varname-tags for archive_command in the docs. Fujii Masao http://git.postgresql.org/pg/commitdiff/4df132895016c6a99355776a8df284ff011a2e94
  • Fix typo in comment. Jim Nasby http://git.postgresql.org/pg/commitdiff/8cc7a4c5fdbe43b9b16b4cf3e07c8115107a8d4e
  • Fix off-by-one error in Assertion. The point of the assertion is to ensure that the arrays allocated in stack are large enough, but the check was one item short. This won't matter in practice because MaxIndexTuplesPerPage is an overestimate, so you can't have that many items on a page in reality. But let's be tidy. Spotted by Anastasia Lubennikova. Backpatch to all supported versions, like the patch that added the assertion. http://git.postgresql.org/pg/commitdiff/b48437d11b9389d724c037385a5cae824d4f8049
  • Collection of typo fixes. Use "a" and "an" correctly, mostly in comments. Two error messages were also fixed (they were just elogs, so no translation work required). Two function comments in pg_proc.h were also fixed. Etsuro Fujita reported one of these, but I found a lot more with grep. Also fix a few other typos spotted while grepping for the a/an typos. For example, "consists out of ..." -> "consists of ...". Plus a "though"/ "through" mixup reported by Euler Taveira. Many of these typos were in old code, which would be nice to backpatch to make future backpatching easier. But much of the code was new, and I didn't feel like crafting separate patches for each branch. So no backpatching. http://git.postgresql.org/pg/commitdiff/4fc72cc7bb9d2105261b8ee45558af50d788cd19
  • Fix more typos in comments. Patch by CharSyam, plus a few more I spotted with grep. http://git.postgresql.org/pg/commitdiff/fa60fb63e511e7bbcf57ce972338711593a5e7c9
  • At promotion, don't leave behind a partial segment on the old timeline. With commit de768844, a copy of the partial segment was archived with the .partial suffix, but the original file was still left in pg_xlog, so it didn't actually solve the problems with archiving the partial segment that it was supposed to solve. With this patch, the partial segment is renamed rather than copied, so we only archive it with the .partial suffix. Also be more robust in detecting if the last segment is already being archived. Previously I used XLogArchiveIsBusy() for that, but that's not quite right. With archive_mode='always', there might be a .ready file for it, and we don't want to rename it to .partial in that case. The old segment is needed until we're fully committed to the new timeline, i.e. until we've written the end-of-recovery WAL record and updated the min recovery point and timeline in the control file. So move the renaming later in the startup sequence, after all that's been done. http://git.postgresql.org/pg/commitdiff/7cbee7c0a1db668c60c020a3fd1e3234daa562a9

Noah Misch a poussé :

  • Permit use of vsprintf() in PostgreSQL code. The next commit needs it. Back-patch to 9.0 (all supported versions). http://git.postgresql.org/pg/commitdiff/cac18a76bb6b08f1ecc2a85e46c9d2ab82dd9d23
  • Check return values of sensitive system library calls. PostgreSQL already checked the vast majority of these, missing this handful that nearly cannot fail. If putenv() failed with ENOMEM in pg_GSS_recvauth(), authentication would proceed with the wrong keytab file. If strftime() returned zero in cache_locale_time(), using the unspecified buffer contents could lead to information exposure or a crash. Back-patch to 9.0 (all supported versions). Other unchecked calls to these functions, especially those in frontend code, pose negligible security concern. This patch does not address them. Nonetheless, it is always better to check return values whose specification provides for indicating an error. In passing, fix an off-by-one error in strftime_win32()'s invocation of WideCharToMultiByte(). Upon retrieving a value of exactly MAX_L10N_DATA bytes, strftime_win32() would overrun the caller's buffer by one byte. MAX_L10N_DATA is chosen to exceed the length of every possible value, so the vulnerable scenario probably does not arise. Security: CVE-2015-3166 http://git.postgresql.org/pg/commitdiff/fd97bd411d1da45b79e63c2124741f8e82cc5a5c
  • Prevent a double free by not reentering be_tls_close(). Reentering this function with the right timing caused a double free, typically crashing the backend. By synchronizing a disconnection with the authentication timeout, an unauthenticated attacker could achieve this somewhat consistently. Call be_tls_close() solely from within proc_exit_prepare(). Back-patch to 9.0 (all supported versions). Benkocs Norbert Attila Security: CVE-2015-3165 http://git.postgresql.org/pg/commitdiff/b0ce385032d72d6acf1e330f733013553fe6affe
  • pgcrypto: Report errant decryption as "Wrong key or corrupt data". This has been the predominant outcome. When the output of decrypting with a wrong key coincidentally resembled an OpenPGP packet header, pgcrypto could instead report "Corrupt data", "Not text data" or "Unsupported compression algorithm". The distinct "Corrupt data" message added no value. The latter two error messages misled when the decrypted payload also exhibited fundamental integrity problems. Worse, error message variance in other systems has enabled cryptologic attacks; see RFC 4880 section "14. Security Considerations". Whether these pgcrypto behaviors are likewise exploitable is unknown. In passing, document that pgcrypto does not resist side-channel attacks. Back-patch to 9.0 (all supported versions). Security: CVE-2015-3167 http://git.postgresql.org/pg/commitdiff/85270ac7a24a50d43ba4bd4d7af1e28b14dee7ee
  • Add error-throwing wrappers for the printf family of functions. All known standard library implementations of these functions can fail with ENOMEM. A caller neglecting to check for failure would experience missing output, information exposure, or a crash. Check return values within wrappers and code, currently just snprintf.c, that bypasses the wrappers. The wrappers do not return after an error, so their callers need not check. Back-patch to 9.0 (all supported versions). Popular free software standard library implementations do take pains to bypass malloc() in simple cases, but they risk ENOMEM for floating point numbers, positional arguments, large field widths, and large precisions. No specification demands such caution, so this commit regards every call to a printf family function as a potential threat. Injecting the wrappers implicitly is a compromise between patch scope and design goals. I would prefer to edit each call site to name a wrapper explicitly. libpq and the ECPG libraries would, ideally, convey errors to the caller rather than abort(). All that would be painfully invasive for a back-patched security fix, hence this compromise. Security: CVE-2015-3166 http://git.postgresql.org/pg/commitdiff/16304a013432931e61e623c8d85e9fe24709d9ba

Robert Haas a poussé :

Andres Freund a poussé :

  • Attach ON CONFLICT SET ... WHERE to the correct planstate. The previous coding was a leftover from attempting to hang all the on conflict logic onto modify table's child nodes. It appears to not have actually caused problems except for explain. Add test exercising the broken and some other code paths. Author: Peter Geoghegan and Andres Freund http://git.postgresql.org/pg/commitdiff/e4942f7a56efcfaabed5db7bde29ee21bef2f6e2
  • Various fixes around ON CONFLICT for rule deparsing. Neither the deparsing of the new alias for INSERT's target table, nor of the inference clause was supported. Also fixup a typo in an error message. Add regression tests to test those code paths. Author: Peter Geoghegan http://git.postgresql.org/pg/commitdiff/9bc77c45199c7d2e525cd5b1457d5a57f6e9edb0
  • Refactor ON CONFLICT index inference parse tree representation. Defer lookup of opfamily and input type of a of a user specified opclass until the optimizer selects among available unique indexes; and store the opclass in the parse analyzed tree instead. The primary reason for doing this is that for rule deparsing it's easier to use the opclass than the previous representation. While at it also rename a variable in the inference code to better fit it's purpose. This is separate from the actual fixes for deparsing to make review easier. http://git.postgresql.org/pg/commitdiff/0740cbd7593d871858c352fab29a59cf7fa54b00
  • Remove the new UPSERT command tag and use INSERT instead. Previously, INSERT with ON CONFLICT DO UPDATE specified used a new command tag -- UPSERT. It was introduced out of concern that INSERT as a command tag would be a misrepresentation for ON CONFLICT DO UPDATE, as some affected rows may actually have been updated. Alvaro Herrera noticed that the implementation of that new command tag was incomplete; in subsequent discussion we concluded that having it doesn't provide benefits that are in line with the compatibility breaks it requires. Catversion bump due to the removal of PlannedStmt->isUpsert. Author: Peter Geoghegan Discussion: 20150520215816.GI5885@postgresql.org http://git.postgresql.org/pg/commitdiff/631d7490074cdaef8026db57a5f2772b8730f600
  • Fix yet another bug in ON CONFLICT rule deparsing. Expand testing of rule deparsing a good bit, it's evidently needed. Author: Peter Geoghegan, Andres Freund Discussion: CAM3SWZQmXxZhQC32QVEOTYfNXJBJ_Q2SDENL7BV14Cq-zL0FLg@mail.gmail.com http://git.postgresql.org/pg/commitdiff/284bef297733e553c73f1c858e0ce1532f754d18

Simon Riggs a poussé :

Andrew Dunstan a poussé :

  • Unpack jbvBinary objects passed to pushJsonbValue pushJsonbValue was accepting jbvBinary objects passed as WJB_ELEM or WJB_VALUE data. While this succeeded, when those objects were later encountered in attempting to convert the result to Jsonb, errors occurred. With this change we ghuarantee that a JSonbValue constructed from calls to pushJsonbValue does not contain any jbvBinary objects. This cures a problem observed with jsonb_delete. This means callers of pushJsonbValue no longer need to perform this unpacking themselves. A subsequent patch will perform some cleanup in that area. The error was not triggered by any 9.4 code, but this is a publicly visible routine, and so the error could be exercised by third party code, therefore backpatch to 9.4. Bug report from Peter Geoghegan, fix by me. http://git.postgresql.org/pg/commitdiff/5302760a50332a684e35b9865ff47ff5fd4970c2

Bruce Momjian a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Álvaro Herrera sent in another revision of a patch to fix some infelicities with MultiXact in recovery.
  • Uriy Zhuravlev sent in a patch to implement ALTER OPERATOR.
  • John Gorman sent in a patch to preserve errno across errmsg() calls.
  • Joshua Drake sent in a doc patch for backups.
  • Jim Nasby sent in another revision of a patch to add an optional parameter to the pg_*_backend functions which allows skipping the backend making the call.
  • Jeff Janes sent in a patch to fix max WAL size in recovery.
  • Alexander Shulgin sent in a patch to generalize the JSON-producing functions in utils/adt/json.c and to provide a set of callbacks which can be overridden the same way that is already provided for parsing JSON.
  • Andrew Gierth sent in a patch to change the type of GROUPING() from int4 to int8.
  • Robert Haas sent in another revision of a patch to allow assessing parallel safety.
  • Alexander Shulgin sent in another revision of a patch to add a --strict-include parameter to pg_dump.
  • Andrew Dunstan sent in a patch to rename jsonb_replace to jsonb_set with a boolean create_missing flag that defaults to false.
  • Amit Kapila sent in another revision of a patch to implement parallel seq scan.
  • Pavel Stehule sent in a patch to allow SET ROLE TO to tab-complete in psql, just as SET ROLE now does.
  • Fabrízio de Royes Mello sent in a patch to fix some spacing in the ALTER TABLE documentation.
  • CharSyam sent in a patch to change magic constants to DEFINE value for readability.
  • Noah Misch sent in a patch to fix some tar name issues.
  • Fabien COELHO sent in another revision of a patch to extend pgbench expressions with functions.
  • Petr Korobeinikov sent in another revision of a patch to add support for \ev viewname and \sv viewname to psql.

par N Bougain le vendredi 29 mai 2015 à 14h34

lundi 25 mai 2015

Dimitri Fontaine

PostgreSQL au JUG de Montpellier

J'ai eu le plaisir de présenter PostgreSQL au Java User Group de Montpellier le 20 mai dernier dans le cadre d'une soirée PostgreSQL is YeSQL. L'accueil réservé par le JUG a été parfait et je tiens à les remercier pour une très bonne soirée de partage et d'échanges autour de PostgreSQL.

Les slides des présentations sont disponibles. Nous avons commencé par une présentation générale sur comment et pourquoi utiliser PostgreSQL en 2015 :

Puis nous avons continué la soirée avec une présentation des Extensions PostgreSQL, dont ce blog regorge d'exemples :

Une série d'articles sont disponibles sur ce blog et reprennent en détails les éléments présentés dans cette deuxième partie :

 

Next month, Postgres Open 2014 is happening in Chicago, and I'll have the pleasure to host a tutorial about PostgreSQL Extensions Writing & Using Postgres Extensions, and a talk aimed at developers wanting to make the best out of PostgreSQL, PostgreSQL for developers:

 

PHP Tour 2014

June, 27 2014

En début de semaine se tenait le PHP Tour 2014 à Lyon, et j'ai eu le privilège d'y être invité afin de présenter comment Utiliser PostgreSQL en 2014.

 

Last week some PostgreSQL users, contributors and advocates have organized a really great conference in Stockholm, Sweden, where I had the please to give the following talk:

 

In our previous article Aggregating NBA data, PostgreSQL vs MongoDB we spent time comparing the pretty new MongoDB Aggregation Framework with the decades old SQL aggregates. Today, let's showcase more of those SQL aggregates, producing a nice histogram right from our SQL console.

 

When reading the article Crunching 30 Years of NBA Data with MongoDB Aggregation I coulnd't help but think that we've been enjoying aggregates in SQL for 3 or 4 decades already. When using PostgreSQL it's even easy to actually add your own aggregates given the SQL command create aggregate.

 

Back from the FODESM 2014 Conference, here's the slides I've been using for the Advanced Extension Use Cases talk I gave, based on the ongoing work to be found under the Tour of Extensions index in this web site.

 

Back From Dublin

November, 05 2013

Last week I had the pleasure to present two talks at the awesome PostgreSQL Conference Europe. The first one was actually a tutorial about Writing & using Postgres Extensions where we spent 3 hours on what are PostgreSQL Extensions, what you can expect from them, and how to develop a new one. Then I also had the opportunity to present the new version of pgloader in a talk about Migrating from MySQL to PostgreSQL.

 

Denormalizing Tags

October, 24 2013

In our Tour of Extensions today's article is about advanced tag indexing. We have a great data collection to play with and our goal today is to be able to quickly find data matching a complex set of tags. So, let's find out those lastfm tracks that are tagged as blues and rhythm and blues, for instance.

 

At the Open World Forum two weeks ago I had the pleasure to meet with Colin Charles. We had a nice talk about the current state of both MariaDB and PostgreSQL, and even were both interviewed by the Open World Forum Team. The interview is now available online. Dear French readers, it's in English.

 

PostgreSQL is an all round impressive Relational DataBase Management System which implements the SQL standard (see the very useful reference page Comparison of different SQL implementations for details). PostgreSQL also provides with unique solutions in the database market and has been leading innovation for some years now. Still, there's no support for Autonomous Transactions within the server itself. Let's have a look at how to easily implement them with PL/Proxy.

 

Let's get back to our Tour of Extensions that had to be kept aside for awhile with other concerns such as last chance PostgreSQL data recovery. Now that we have a data loading tool up to the task (read about it in the Loading Geolocation Data article) we're going to be able to play with the awesome ip4r extension from RhodiumToad.

 

Last Friday I had the chance to be speaking at the Open World Forum in the NewSQL track, where we had lots of interest and excitement around the NoSQL offerings. Of course, my talk was about explaining how PostgreSQL is Web Scale with some historical background and technical examples about what this database engine is currently capable of.

 

Using trigrams against typos

September, 06 2013

In our ongoing Tour of Extensions we played with earth distance in How far is the nearest pub? then with hstore in a series about trigger, first to generalize Trigger Parameters then to enable us to Auditing Changes with Hstore. Today we are going to work with pg_trgm which is the trigrams PostgreSQL extension: its usage got seriously enhanced in recent PostgreSQL releases and it's now a poor's man Full Text Search engine.

 

In a previous article about Trigger Parameters we have been using the extension hstore in order to compute some extra field in our records, where the fields used both for the computation and for storing the results were passed in as dynamic parameters. Today we're going to see another trigger use case for hstore: we are going to record changes made to our tuples.

 

Trigger Parameters

August, 23 2013

Sometimes you want to compute values automatically at INSERT time, like for example a duration column out of a start and an end column, both timestamptz. It's easy enough to do with a BEFORE TRIGGER on your table. What's more complex is to come up with a parametrized spelling of the trigger, where you can attach the same stored procedure to any table even when the column names are different from one another.

 

There was SQL before window functions and SQL after window functions: that's how powerful this tool is. Being that of a deal breaker unfortunately means that it can be quite hard to grasp the feature. This article aims at making it crystal clear so that you can begin using it today and are able to reason about it and recognize cases where you want to be using window functions.

 

In our recent article about The Most Popular Pub Names we did have a look at how to find the pubs nearby, but didn't compute the distance in between that pub and us. That's because how to compute a distance given a position on the earth expressed as longitude and latitude is not that easy. Today, we are going to solve that problem nonetheless, thanks to PostgreSQL Extensions.

 

In his article titled The Most Popular Pub Names Ross Lawley did show us how to perform some quite interesting geographic queries against MongoDB, using some nice Open Data found at the Open Street Map project.

 

In a recent article here we've been talking about how do do Batch Updates in a very efficient way, using the Writable CTE features available in PostgreSQL 9.1. I sometime read how Common Table Expressions changed the life of fellow DBAs and developers, and would say that Writable CTE are at least the same boost again.

 

In a recent article Craig Kerstiens from Heroku did demo the really useful crosstab extension. That function allows you to pivot a table so that you can see the data from different categories in separate columns in the same row rather than in separate rows. The article from Craig is Pivoting in Postgres.

 

Tonight I had the pleasure to present a talk at the Dublin PostgreSQL User Group using remote technologies. The talk is about how to make the most ouf of PostgreSQL when using SQL as a developer, and tries to convince you to dive into mastering SQL by showing how to solve an application example all in SQL, using window functions and common table expressions.

 

Nearest Big City

May, 02 2013

In this article, we want to find the town with the greatest number of inhabitants near a given location.

 

The Need For Speed

March, 29 2013

Hier se tenait la cinquième édition de la conférence organisée par dalibo, où des intervenants extérieurs sont régulièrement invités. Le thème hier était à la fois clair et très vaste : la performance.

 

Batch Update

March, 15 2013

Performance consulting involves some tricks that you have to teach over and over again. One of them is that SQL tends to be so much better at dealing with plenty of rows in a single statement when compared to running as many statements, each one against a single row.

 

HyperLogLog Unions

February, 26 2013

In the article from yesterday we talked about PostgreSQL HyperLogLog with some details. The real magic of that extension has been skimmed over though, and needs another very small article all by itself, in case you missed it.

 

PostgreSQL HyperLogLog

February, 25 2013

If you've been following along at home the newer statistics developments, you might have heard about this new State of The Art Cardinality Estimation Algorithm called HyperLogLog. This technique is now available for PostgreSQL in the extension postgresql-hll available at https://github.com/aggregateknowledge/postgresql-hll and soon to be in debian.

 

PostgreSQL for developers

November, 02 2012

As Guillaume says, we've been enjoying a great evening conference in Lyon 2 days ago, presenting PostgreSQL to developers. He did the first hour presenting the project and the main things you want to know to start using PostgreSQL in production, then I took the opportunity to be talking to developers to show off some SQL.

 

Reset Counter

October, 05 2012

I've been given a nice puzzle that I think is a good blog article opportunity, as it involves some thinking and window functions.

 

Let's say you need to ALTER TABLE foo ALTER COLUMN bar TYPE bigint;, and PostgreSQL is helpfully telling you that no you can't because such and such views depend on the column. The basic way to deal with that is to copy paste from the error message the names of the views involved, then prepare a script wherein you first DROP VIEW ...; then ALTER TABLE and finally CREATE VIEW again, all in the same transaction.

 

Dynamic Triggers in PLpgSQL

November, 24 2010

You certainly know that implementing dynamic triggers in PLpgSQL is impossible. But I had a very bad night, being up from as soon as 3:30 am today, so that when a developer asked me about reusing the same trigger function code from more than one table and for a dynamic column name, I didn't remember about it being impossible.

 

The drawback of hosting a static only website is, obviously, the lack of comments. What happens actually, though, is that I receive very few comments by direct mail. As I don't get another spam source to cleanup, I'm left unconvinced that's such a drawback. I still miss the low probability of seeing blog readers exchange directly, but I think a tapoueh.org mailing list would be my answer, here...

 

Window Functions example

September, 09 2010

So, when 8.4 came out there was all those comments about how getting window functions was an awesome addition. Now, it seems that a lot of people seeking for help in #postgresql just don't know what kind of problem this feature helps solving. I've already been using them in some cases here in this blog, for getting some nice overview about Partitioning: relation size per “group”.

 

Happy Numbers

August, 30 2010

After discovering the excellent Gwene service, which allows you to subscribe to newsgroups to read RSS content ( blogs, planets, commits, etc), I came to read this nice article about Happy Numbers. That's a little problem that fits well an interview style question, so I first solved it yesterday evening in Emacs Lisp as that's the language I use the most those days.

 

Playing with bit strings

August, 26 2010

The idea of the day ain't directly from me, I'm just helping with a very thin subpart of the problem. The problem, I can't say much about, let's just assume you want to reduce the storage of MD5 in your database, so you want to abuse bit strings. A solution to use them works fine, but the datatype is still missing some facilities, for example going from and to hexadecimal representation in text.

 

This time, we are trying to figure out where is the bulk of the data on disk. The trick is that we're using DDL partitioning, but we want a “nice” view of size per partition set. Meaning that if you have for example a parent table foo with partitions foo_201006 and foo_201007, you would want to see a single category foo containing the accumulated size of all the partitions underneath foo.

 

This time we're having a database where sequences were used, but not systematically as a default value of a given column. It's mainly an historic bad idea, but you know the usual excuse with bad ideas and bad code: the first 6 months it's experimental, after that it's historic.

 

The problem was raised this week on IRC and this time again I felt it would be a good occasion for a blog entry: how to load an XML file content into a single field?

par dim@tapoueh.org (Dimitri Fontaine) le lundi 25 mai 2015 à 15h17

vendredi 22 mai 2015

Actualités PostgreSQL.fr

Nouvelles versions mineures

Le PostgreSQL Global Development Group a publié un correctif qui concerne de nombreuses fonctionnalités et failles de sécurité. Toutes les versions de PostgreSQL sont concernées. Les nouvelles versions sont 9.4.2, 9.3.7, 9.2.11, 9.1.16 et 9.0.20. La mise à jour contient le correctif d'une possible corruption de données sur les versions 9.3 et 9.4 de PostgreSQL. Il est recommandé de procéder à la mise à jour dès que possible.

Correctif de la corruption de données

Les utilisateurs des versions 9.3 et 9.4 sont concernés par ce correctif. Les versions 9.2 ou précédentes ne sont pas affectées. La mise à jour corrige le problème du bouclage d'identifiant « multixact » ("multixact wraparound"), qui induit une corruption ou une perte de données. Les utilisateurs dont le taux de transactions est très élevé (1 million par heure) sur des bases ayant de nombreuses clés étrangères sont particulièrement vulnérables. Il est plus que recommandé de procéder aux mises à jour des installations dans les prochains jours.

Correctifs de sécurité

Cette mise à jour corrige trois failles de sécurité rapportées ces derniers mois. Aucune n'est particulièrement urgente. Toutefois, il est conseillé aux utilisateurs de vérifier si leurs installations sont concernées.

   CVE-2015-3165 Double libération de mémoire après timeout d'authentification.
   CVE-2015-3166 Erreurs imprévues de la bibliothèque standard
   CVE-2015-3167 pgcrypto contient plusieurs messages d'erreur lors de déchiffrement avec une mauvaise clé.

Il est également conseillé aux utilisateurs des authentifications par Kerberos, GSSAPI, ou SSPI de positionner include_realm à 1 dans le fichier pg_hba.conf, cette valeur devenant la valeur par défaut dans les futures versions.

Pour plus d'informations sur ces failles, on peut se référer aux pages de sécurité sur le site de PostgreSQL.

Autres correctifs et améliorations

Une nouvelle version, qui n'est pas celle par défaut, de l'extension citext corrige les fonctions regexp_matches() précédemment non-documentées, pour les aligner sur les versions texte usuelles de ces fonctions. La version corrigée utilisant un type de retour différent de l'ancienne, il est nécessaire de tester les applications avant d'exécuter la commande "ALTER EXTENSION citext UPDATE".

Plus de 50 autres correctifs sont fournis par ces mises à jour. La plupart concernent l'ensemble des versions supportées.

On peut citer :

 * la traduction des dates et estampilles temporelles infinies en "infinity" lors de conversion en json ;
 * la correction des fonctions json/jsonb populate_record() et to_record() ;
 * la correction de la vérification des contraintes d'exclusion différée ;
 * l'amélioration de la planification des requêtes touchant plusieurs schémas ;
 * la correction de trois problèmes de jointures ;
 * la garantie de verrouillage correct lors de l'utilisation de barrières de sécurité ("security barriers") dans les vues ;
 * la correction du verrou mort au démarrage lorsque le paramètre max_prepared_transactions est trop faible ;
 * le parcours récusif par fsync() du répertoire de données après un crash ;
 * la correction du lanceur d'autovacuum qui pouvait ne pas s'arrêter ;
 * la gestion de signaux inattendus dans la fonction LockBufferForCleanup() ;
 * la correction du crash de COPY IN vers une table possédant des contraintes de vérification ;
 * la suppression de l'attente de réplication synchrone sur les transactions en lecture seule ;
 * la correction de deux problèmes sur les index hash ;
 * la suppression des fuites mémoire lors de vacuum sur les index GIN ; 
 * la correction de deux problèmes sur les « background workers » ;
 * plusieurs corrections sur la réplication par décodage logique ;
 * plusieurs corrections sur pg_dump et pg_ugrade.

Cette version inclut une mise à jour vers la version 2015d de tzdata, avec les mise à jour pour l'Egypte, la Mongolie, la Palestine, et des modifications historiques pour le Canada et le Chili.

Fin de vie proche pour la branche 9.0

La version 9.0 arrive en fin de vie en septembre 2015. Il est probable que cette mise à jour soit une des dernières de cette branche. Les utilisateurs de PostgreSQL 9.0 peuvent commencer à planifier la mise à jour vers une version plus récente. Voir la page d'information sur les versions (http://www.postgresql.org/support/versioning ) pour de plus amples informations sur les dates de fin de vie.

Comme pour les autres versions mineures, il n'est pas nécessaire d'extraire et recharger les bases ou d'utiliser pg_upgrade pour appliquer cette mise à jour. Il suffit d'arrêter PostgreSQL, et de mettre à jour les binaires.

Les utilisateurs de l'extension citext auront une commande à jouer.

Les utilisateurs ayant négligé les mises à jour précédentes auront peut-être quelques actions post-mise à jour à effectuer.

Liens :

 * Téléchargement
 * Notes de version
 * Page sécurité

par SAS le vendredi 22 mai 2015 à 13h44

lundi 18 mai 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 17 mai 2015

Sur Vimeo se trouve un canal italophone dédié à PostgreSQL : https://vimeo.com/user10626375/videos

La CLEI (conférence latino-américaine d'informatique) se tiendra à Arequipa (Pérou) du 19 au 23 octobre 2015 : https://sites.google.com/a/spc.org.pe/clei2015/

Informatica 2016, qui aura lieu à Cuba, recherche du contenu touchant à PostgreSQL pour ses ateliers open-source : http://www.informaticahabana.cu/

PostgreSQL Conference Europe 2015 aura lieu du 27 au 30 octobre au Vienna Marriott Hotel à Vienne (Autriche). L'appel à conférenciers est lancé jusqu'au 7 août : http://2015.pgconf.eu/

Le SwissPUG est à présent ouvert aux nouveaux membres : http://www.swisspug.org/wiki/index.php/Special:Registration

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Robert Haas a poussé :

  • Fix DetermineSafeOldestOffset for the case where there are no mxacts. Commit b69bf30b9bfacafc733a9ba77c9587cf54d06c0c failed to take into account the possibility that there might be no multixacts in existence at all. Report by Thomas Munro; patch by me. http://git.postgresql.org/pg/commitdiff/312747c224be4c0a880b5172a010e980f8b654ca
  • Advance the stop point for multixact offset creation only at checkpoint. Commit b69bf30b9bfacafc733a9ba77c9587cf54d06c0c advanced the stop point at vacuum time, but this has subsequently been shown to be unsafe as a result of analysis by myself and Thomas Munro and testing by Thomas Munro. The crux of the problem is that the SLRU deletion logic may get confused about what to remove if, at exactly the right time during the checkpoint process, the head of the SLRU crosses what used to be the tail. This patch, by me, fixes the problem by advancing the stop point only following a checkpoint. This has the additional advantage of making the removal logic work during recovery more like the way it works during normal running, which is probably good. At least one of the calls to DetermineSafeOldestOffset which this patch removes was already dead, because MultiXactAdvanceOldest is called only during recovery and DetermineSafeOldestOffset was set up to do nothing during recovery. That, however, is inconsistent with the principle that recovery and normal running should work similarly, and was confusing to boot. Along the way, fix some comments that previous patches in this area neglected to update. It's not clear to me whether there's any concrete basis for the decision to use only half of the multixact ID space, but it's neither necessary nor sufficient to prevent multixact member wraparound, so the comments should not say otherwise. http://git.postgresql.org/pg/commitdiff/f6a6c46d7fd72878d37c75d4a3215d5a62128d0b
  • Even when autovacuum=off, force it for members as we do in other cases. Thomas Munro, with some adjustments by me. http://git.postgresql.org/pg/commitdiff/04e6d3b877e060d8445eb653b7ea26b1ee5cec6b
  • Increase threshold for multixact member emergency autovac to 50%. Analysis by Noah Misch shows that the 25% threshold set by commit 53bb309d2d5a9432d2602c93ed18e58bd2924e15 is lower than any other, similar autovac threshold. While we don't know exactly what value will be optimal for all users, it is better to err a little on the high side than on the low side. A higher value increases the risk that users might exhaust the available space and start seeing errors before autovacuum can clean things up sufficiently, but a user who hits that problem can compensate for it by reducing autovacuum_multixact_freeze_max_age to a value dependent on their average multixact size. On the flip side, if the emergency cap imposed by that patch kicks in too early, the user will experience excessive wraparound scanning and will be unable to mitigate that problem by configuration. The new value will hopefully reduce the risk of such bad experiences while still providing enough headroom to avoid multixact member exhaustion for most users. Along the way, adjust the documentation to reflect the effects of commit 04e6d3b877e060d8445eb653b7ea26b1ee5cec6b, which taught autovacuum to run for multixact wraparound even when autovacuum is configured off. http://git.postgresql.org/pg/commitdiff/b4d4ce1d50bbdf82cd2e2c1c7172b936df01c01d
  • Remove useless assertion. Here, snapshot->xcnt is an unsigned type, so it will always be non-negative. http://git.postgresql.org/pg/commitdiff/ae6157164faf5ec1636a9acfe18bfd28a31db098
  • Extend abbreviated key infrastructure to datum tuplesorts. Andrew Gierth, reviewed by Peter Geoghegan and by me. http://git.postgresql.org/pg/commitdiff/78efd5c1edb59017f06ef96773e64e6539bfbc86
  • Fix comment. Commit 78efd5c1edb59017f06ef96773e64e6539bfbc86 overlooked this. Report by Peter Geoghegan. http://git.postgresql.org/pg/commitdiff/61f68e0bed63aa5090e8be7c912843e49b30fc1e
  • doc: CREATE FOREIGN TABLE now allows CHECK ( ... ) NO INHERIT. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/92edba2665ae7bf43ed03538311e63652f9e2373

Bruce Momjian a poussé :

Tom Lane a poussé :

  • Fix incorrect checking of deferred exclusion constraint after a HOT update. If a row that potentially violates a deferred exclusion constraint is HOT-updated later in the same transaction, the exclusion constraint would be reported as violated when the check finally occurs, even if the row(s) the new row originally conflicted with have since been removed. This happened because the wrong TID was passed to check_exclusion_constraint(), causing the live HOT-updated row to be seen as a conflicting row rather than recognized as the row-under-test. Per bug #13148 from Evan Martin. It's been broken since exclusion constraints were invented, so back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/20781765f77c1fb6465aba97d211636ce92e7a0e
  • Add support for doing late row locking in FDWs. Previously, FDWs could only do "early row locking", that is lock a row as soon as it's fetched, even though local restriction/join conditions might discard the row later. This patch adds callbacks that allow FDWs to do late locking in the same way that it's done for regular tables. To make use of this feature, an FDW must support the "ctid" column as a unique row identifier. Currently, since ctid has to be of type TID, the feature is of limited use, though in principle it could be used by postgres_fdw. We may eventually allow FDWs to specify another data type for ctid, which would make it possible for more FDWs to use this feature. This commit does not modify postgres_fdw to use late locking. We've tested some prototype code for that, but it's not in committable shape, and besides it's quite unclear whether it actually makes sense to do late locking against a remote server. The extra round trips required are likely to outweigh any benefit from improved concurrency. Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me http://git.postgresql.org/pg/commitdiff/afb9249d06f47d7a6d4a89fea0c3625fe43c5a5d
  • Fix postgres_fdw to return the right ctid value in EvalPlanQual cases. If a postgres_fdw foreign table is a non-locked source relation in an UPDATE, DELETE, or SELECT FOR UPDATE/SHARE, and the query selects its ctid column, the wrong value would be returned if an EvalPlanQual recheck occurred. This happened because the foreign table's result row was copied via the ROW_MARK_COPY code path, and EvalPlanQualFetchRowMarks just unconditionally set the reconstructed tuple's t_self to "invalid". To fix that, we can have EvalPlanQualFetchRowMarks copy the composite datum's t_ctid field, and be sure to initialize that along with t_self when postgres_fdw constructs a tuple to return. If we just did that much then EvalPlanQualFetchRowMarks would start returning "(0,0)" as ctid for all other ROW_MARK_COPY cases, which perhaps does not matter much, but then again maybe it might. The cause of that is that heap_form_tuple, which is the ultimate source of all composite datums, simply leaves t_ctid as zeroes in newly constructed tuples. That seems like a bad idea on general principles: a field that's really not been initialized shouldn't appear to have a valid value. So let's eat the trivial additional overhead of doing "ItemPointerSetInvalid(&(td->t_ctid))" in heap_form_tuple. This closes out our handling of Etsuro Fujita's report that tableoid and ctid weren't correctly set in postgres_fdw EvalPlanQual cases. Along the way we did a great deal of work to improve FDWs' ability to control row locking behavior; which was not wasted effort by any means, but it didn't end up being a fix for this problem because that feature would be too expensive for postgres_fdw to use all the time. Although the fix for the tableoid misbehavior was back-patched, I'm hesitant to do so here; it seems far less likely that people would care about remote ctid than tableoid, and even such a minor behavioral change as this in heap_form_tuple is perhaps best not back-patched. So commit to HEAD only, at least for the moment. Etsuro Fujita, with some adjustments by me http://git.postgresql.org/pg/commitdiff/0bb8528b5c738b45d0b65844750588c16bf75c52
  • Fix distclean/maintainer-clean targets to remove top-level tmp_install dir. The top-level makefile removes tmp_install in its "clean" target, but the distclean and maintainer-clean targets overlooked that (and they don't simply invoke clean, because that would result in an extra tree traversal). While at it, let's just make sure that removing GNUmakefile itself is the very last step of the recipe. http://git.postgresql.org/pg/commitdiff/9660710e2f5bbbf1b18640fbc5edcceafe7a10ad
  • Support "expanded" objects, particularly arrays, for better performance. This patch introduces the ability for complex datatypes to have an in-memory representation that is different from their on-disk format. On-disk formats are typically optimized for minimal size, and in any case they can't contain pointers, so they are often not well-suited for computation. Now a datatype can invent an "expanded" in-memory format that is better suited for its operations, and then pass that around among the C functions that operate on the datatype. There are also provisions (rudimentary as yet) to allow an expanded object to be modified in-place under suitable conditions, so that operations like assignment to an element of an array need not involve copying the entire array. The initial application for this feature is arrays, but it is not hard to foresee using it for other container types like JSON, XML and hstore. I have hopes that it will be useful to PostGIS as well. In this initial implementation, a few heuristics have been hard-wired into plpgsql to improve performance for arrays that are stored in plpgsql variables. We would like to generalize those hacks so that other datatypes can obtain similar improvements, but figuring out some appropriate APIs is left as a task for future work. (The heuristics themselves are probably not optimal yet, either, as they sometimes force expansion of arrays that would be better left alone.) Preliminary performance testing shows impressive speed gains for plpgsql functions that do element-by-element access or update of large arrays. There are other cases that get a little slower, as a result of added array format conversions; but we can hope to improve anything that's annoyingly bad. In any case most applications should see a net win. Tom Lane, reviewed by Andres Freund http://git.postgresql.org/pg/commitdiff/1dc5ebc9077ab742079ce5dac9a6664248d42916
  • Suppress uninitialized-variable warning. http://git.postgresql.org/pg/commitdiff/6c9e93d3ffbf99435636103ad69d6469c64e2aef
  • Docs: fix erroneous claim about max byte length of GB18030. This encoding has characters up to 4 bytes long, not 2. http://git.postgresql.org/pg/commitdiff/333d0779627a6c6125c018ea39da4427d3bdd93f
  • Fix portability issue in pg_audit. "%ld" is not a portable way to print int64's. This may explain the buildfarm crashes we're seeing --- it seems to make dromedary happy, at least. http://git.postgresql.org/pg/commitdiff/35a1e1d1593f4355c9d87bbc8208a8736801a607
  • Teach UtfToLocal/LocalToUtf to support algorithmic encoding conversions. Until now, these functions have only supported encoding conversions using lookup tables, which is fine as long as there's not too many code points to convert. However, GB18030 expects all 1.1 million Unicode code points to be convertible, which would require a ridiculously-sized lookup table. Fortunately, a large fraction of those conversions can be expressed through arithmetic, ie the conversions are one-to-one in certain defined ranges. To support that, provide a callback function that is used after consulting the lookup tables. (This patch doesn't actually change anything about the GB18030 conversion behavior, just provide infrastructure for fixing it.) Since this requires changing the APIs of UtfToLocal/LocalToUtf anyway, take the opportunity to rearrange their argument lists into what seems to me a saner order. And beautify the call sites by using lengthof() instead of error-prone sizeof() arithmetic. In passing, also mark all the lookup tables used by these calls "const". This moves an impressive amount of stuff into the text segment, at least on my machine, and is safer anyhow. http://git.postgresql.org/pg/commitdiff/7730f48ede0d222e7f750541d3d5f0f74d75d99b
  • Honor traditional SGML NAMELEN limit. We've conformed to this limit in the past, so might as well continue to. Aaron Swenson http://git.postgresql.org/pg/commitdiff/4b8f797f672bef07b4e87b4650b4035731b61d84
  • Fix insufficiently-paranoid GB18030 encoding verifier. The previous coding effectively only verified that the second byte of a multibyte character was in the expected range; moreover, it wasn't careful to make sure that the second byte even exists in the buffer before touching it. The latter seems unlikely to cause any real problems in the field (in particular, it could never be a problem with null-terminated input), but it's still a bug. Since GB18030 is not a supported backend encoding, the only thing we'd really be doing with GB18030 text is converting it to UTF8 in LocalToUtf, which would fail anyway on any invalid character for lack of a match in its lookup table. So the only user-visible consequence of this change should be that you'll get "invalid byte sequence for encoding" rather than "character has no equivalent" for malformed GB18030 input. However, impending changes to the GB18030 conversion code will require these tighter up-front checks to avoid producing bogus results. http://git.postgresql.org/pg/commitdiff/a868931fecdf93f3ceb1c9431bb93757b706269d
  • Fix outdated src/test/mb/ tests, and add a GB18030 test. The expected-output files for these tests were broken by the recent addition of a warning for hash indexes. Update them. Also add a test case for GB18030 encoding, similar to the other ones. This is a pretty weak test, but it's better than nothing. http://git.postgresql.org/pg/commitdiff/07af523870bcfe930134054febd3a6a114942e5b
  • Extend GB18030 encoding conversion to cover full Unicode range. Our previous code for GB18030 <-> UTF8 conversion only covered Unicode code points up to U+FFFF, but the actual spec defines conversions for all code points up to U+10FFFF. That would be rather impractical as a lookup table, but fortunately there is a simple algorithmic conversion between the additional code points and the equivalent GB18030 byte patterns. Make use of the just-added callback facility in LocalToUtf/UtfToLocal to perform the additional conversions. Having created the infrastructure to do that, we can use the same code to map certain linearly-related subranges of the Unicode space below U+FFFF, allowing removal of the corresponding lookup table entries. This more than halves the lookup table size, which is a substantial savings; utf8_and_gb18030.so drops from nearly a megabyte to about half that. In support of doing that, replace ISO10646-GB18030.TXT with the data file gb-18030-2000.xml (retrieved from http://source.icu-project.org/repos/icu/data/trunk/charset/data/xml/ ) in which these subranges have been deleted from the simple lookup entries. Per bug #12845 from Arjen Nienhuis. The conversion code added here is based on his proposed patch, though I whacked it around rather heavily. http://git.postgresql.org/pg/commitdiff/8d3e0906df5496b853cc763f87b9ffd2ae27adbe
  • Fix uninitialized variable. Per compiler warnings. http://git.postgresql.org/pg/commitdiff/66493dd7aa02e19e93f7d5687acaab70075db34f
  • Improve test for CONVERT() with GB18030 <-> UTF8. Add a bit of coverage of high code points. Arjen Nienhuis http://git.postgresql.org/pg/commitdiff/199f5973c50fe94e128508ff2218c42126fd0ee1
  • Update time zone data files to tzdata release 2015d. DST law changes in Egypt, Mongolia, Palestine. Historical corrections for Canada and Chile. Revised zone abbreviation for America/Adak (HST/HDT not HAST/HADT). http://git.postgresql.org/pg/commitdiff/9d366c1f3d758c3b80cd482a6d4528960c4fc325
  • Avoid direct use of INFINITY. It's not very portable. Per buildfarm. http://git.postgresql.org/pg/commitdiff/12cc299c65ba6124c7d459f2605404ad43909db3
  • More portability fixing for bipartite_match.c. <float.h> is required for isinf() on some platforms. Per buildfarm. http://git.postgresql.org/pg/commitdiff/26058bf0dc5fca0a5107b2ee136a81a86cf36049
  • Fix docs typo. I don't think "respectfully" is what was meant here... http://git.postgresql.org/pg/commitdiff/c65aa7a87e4232d7eefea3d78ec7e1d9f8b4708b
  • First-draft release notes for 9.4.2 et al. As usual, the release notes for older branches will be made by cutting these down, but put them up for community review first. http://git.postgresql.org/pg/commitdiff/0563b4c0c3b7cc2323cfb63e11d723764e2d5f7d
  • Release notes for 9.4.2, 9.3.7, 9.2.11, 9.1.16, 9.0.20. http://git.postgresql.org/pg/commitdiff/a0891d2d0136ea06cde957635338c0c238df87de

Stephen Frost a poussé :

Peter Eisentraut a poussé :

Álvaro Herrera a poussé :

  • "Fix" test_ddl_deparse regress test schedule. MSVC is not smart enough to figure it out, so dumb down the Makefile and remove the schedule file. Also add a .gitignore file. Author: Michael Paquier http://git.postgresql.org/pg/commitdiff/007c932e5aaf6d68087f134b8557bbb7db149e94
  • Allow on-the-fly capture of DDL event details. This feature lets user code inspect and take action on DDL events. Whenever a ddl_command_end event trigger is installed, DDL actions executed are saved to a list which can be inspected during execution of a function attached to ddl_command_end. The set-returning function pg_event_trigger_ddl_commands can be used to list actions so captured; it returns data about the type of command executed, as well as the affected object. This is sufficient for many uses of this feature. For the cases where it is not, we also provide a "command" column of a new pseudo-type pg_ddl_command, which is a pointer to a C structure that can be accessed by C code. The struct contains all the info necessary to completely inspect and even reconstruct the executed command. There is no actual deparse code here; that's expected to come later. What we have is enough infrastructure that the deparsing can be done in an external extension. The intention is that we will add some deparsing code in a later release, as an in-core extension. A new test module is included. It's probably insufficient as is, but it should be sufficient as a starting point for a more complete and future-proof approach. Authors: Álvaro Herrera, with some help from Andres Freund, Ian Barwick, Abhijit Menon-Sen. Reviews by Andres Freund, Robert Haas, Amit Kapila, Michael Paquier, Craig Ringer, David Steele. Additional input from Chris Browne, Dimitri Fontaine, Stephen Frost, Petr Jelínek, Tom Lane, Jim Nasby, Steven Singer, Pavel Stěhule. Based on original work by Dimitri Fontaine, though I didn't use his code. Discussion: https://www.postgresql.org/message-id/m2txrsdzxa.fsf@2ndQuadrant.fr https://www.postgresql.org/message-id/20131108153322.GU5809@eldon.alvh.no-ip.org https://www.postgresql.org/message-id/20150215044814.GL3391@alvh.no-ip.org http://git.postgresql.org/pg/commitdiff/b488c580aef4e05f39be5daaab6464da5b22a494
  • Move strategy numbers to include/access/stratnum.h. For upcoming BRIN opclasses, it's convenient to have strategy numbers defined in a single place. Since there's nothing appropriate, create it. The StrategyNumber typedef now lives there, as well as existing strategy numbers for B-trees (from skey.h) and R-tree-and-friends (from gist.h). skey.h is forced to include stratnum.h because of the StrategyNumber typedef, but gist.h is not; extensions that currently rely on gist.h for rtree strategy numbers might need to add a new A few .c files can stop including skey.h and/or gist.h, which is a nice side benefit. Per discussion: https://www.postgresql.org/message-id/20150514232132.GZ2523@alvh.no-ip.org Authored by Emre Hasegeli and Álvaro. (It's not clear to me why bootscanner.l has any #include lines at all.) http://git.postgresql.org/pg/commitdiff/26df7066cc229887d4defdf1d105c0a22b8a88fb
  • Add BRIN infrastructure for "inclusion" opclasses. This lets BRIN be used with R-Tree-like indexing strategies. Also provided are operator classes for range types, box and inet/cidr. The infrastructure provided here should be sufficient to create operator classes for similar datatypes; for instance, opclasses for PostGIS geometries should be doable, though we didn't try to implement one. (A box/point opclass was also submitted, but we ripped it out before commit because the handling of floating point comparisons in existing code is inconsistent and would generate corrupt indexes.) Author: Emre Hasegeli. Cosmetic changes by me. Review: Andreas Karlsson http://git.postgresql.org/pg/commitdiff/b0b7be61337fc64147f2ad0af5bf2c0e6b8a709f

Andrew Dunstan a poussé :

  • pg_basebackup -F t now succeeds with a long symlink target http://git.postgresql.org/pg/commitdiff/97e0aa697983cf7f7f79e69f2dc248fdefb7dbf6
  • Map basebackup tablespaces using a tablespace_map file. Windows can't reliably restore symbolic links from a tar format, so instead during backup start we create a tablespace_map file, which is used by the restoring postgres to create the correct links in pg_tblspc. The backup protocol also now has an option to request this file to be included in the backup stream, and this is used by pg_basebackup when operating in tar mode. This is done on all platforms, not just Windows. This means that pg_basebackup will not not work in tar mode against 9.4 and older servers, as this protocol option isn't implemented there. Amit Kapila, reviewed by Dilip Kumar, with a little editing from me. http://git.postgresql.org/pg/commitdiff/72d422a5227ef6f76f412486a395aba9f53bf3f0
  • Fix some errors from jsonb functions patch. The catalog version should have been bumped, and the alternative regression result file was not up to date with the name of jsonb_pretty. http://git.postgresql.org/pg/commitdiff/5c7df74204e2fb9440b576518d40fcf3ac65c8ac
  • Fix jsonb replace and delete on scalars and empty structures. These operations now error out if attempted on scalars, and simply return the input if attempted on empty arrays or objects. Along the way we remove the unnecessary cloning of the input when it's known to be unchanged. Regression tests covering these cases are added. http://git.postgresql.org/pg/commitdiff/3f2cec797ecceb7467e365410506c0817f9d0163
  • Additional functions and operators for jsonb. jsonb_pretty(jsonb) produces nicely indented json output. jsonb || jsonb concatenates two jsonb values. jsonb - text removes a key and its associated value from the json jsonb - int removes the designated array element jsonb - text[] removes a key and associated value or array element at the designated path jsonb_replace(jsonb,text[],jsonb) replaces the array element designated by the path or the value associated with the key designated by the path with the given value. Original work by Dmitry Dolgov, adapted and reworked for PostgreSQL core by Andrew Dunstan, reviewed and tidied up by Petr Jelinek. http://git.postgresql.org/pg/commitdiff/c6947010ceb42143d9f047c65c1eac2b38928ab7

Andres Freund a poussé :

  • Fix ON CONFLICT bugs that manifest when used in rules. Specifically the tlist and rti of the pseudo "excluded" relation weren't properly treated by expression_tree_walker, which lead to errors when excluded was referenced inside a rule because the varnos where not properly adjusted. Similar omissions in OffsetVarNodes and expression_tree_mutator had less impact, but should obviously be fixed nonetheless. A couple tests of for ON CONFLICT UPDATE into INSERT rule bearing relations have been added. In passing I updated a couple comments. http://git.postgresql.org/pg/commitdiff/4af6e61a363246cf7fff3368a76603b0ce9945dd
  • Add pgstattuple_approx() to the pgstattuple extension. The new function allows to estimate bloat and other table level statics in a faster, but approximate, way. It does so by using information from the free space map for pages marked as all visible in the visibility map. The rest of the table is actually read and free space/bloat is measured accurately. In many cases that allows to get bloat information much quicker, causing less IO. Author: Abhijit Menon-Sen Reviewed-By: Andres Freund, Amit Kapila and Tomas Vondra Discussion: 20140402214144.GA28681@kea.toroid.org http://git.postgresql.org/pg/commitdiff/5850b20f58a594ac69f4f77b24cad94fc3bfd946
  • Support GROUPING SETS, CUBE and ROLLUP. This SQL standard functionality allows to aggregate data by different GROUP BY clauses at once. Each grouping set returns rows with columns grouped by in other sets set to NULL. This could previously be achieved by doing each grouping as a separate query, conjoined by UNION ALLs. Besides being considerably more concise, grouping sets will in many cases be faster, requiring only one scan over the underlying data. The current implementation of grouping sets only supports using sorting for input. Individual sets that share a sort order are computed in one pass. If there are sets that don't share a sort order, additional sort & aggregation steps are performed. These additional passes are sourced by the previous sort step; thus avoiding repeated scans of the source data. The code is structured in a way that adding support for purely using hash aggregation or a mix of hashing and sorting is possible. Sorting was chosen to be supported first, as it is the most generic method of implementation. Instead of, as in an earlier versions of the patch, representing the chain of sort and aggregation steps as full blown planner and executor nodes, all but the first sort are performed inside the aggregation node itself. This avoids the need to do some unusual gymnastics to handle having to return aggregated and non-aggregated tuples from underlying nodes, as well as having to shut down underlying nodes early to limit memory usage. The optimizer still builds Sort/Agg node to describe each phase, but they're not part of the plan tree, but instead additional data for the aggregation node. They're a convenient and preexisting way to describe aggregation and sorting. The first (and possibly only) sort step is still performed as a separate execution step. That retains similarity with existing group by plans, makes rescans fairly simple, avoids very deep plans (leading to slow explains) and easily allows to avoid the sorting step if the underlying data is sorted by other means. A somewhat ugly side of this patch is having to deal with a grammar ambiguity between the new CUBE keyword and the cube extension/functions named cube (and rollup). To avoid breaking existing deployments of the cube extension it has not been renamed, neither has cube been made a reserved keyword. Instead precedence hacking is used to make GROUP BY cube(..) refer to the CUBE grouping sets feature, and not the function cube(). To actually group by a function cube(), unlikely as that might be, the function name has to be quoted. Needs a catversion bump because stored rules may change. Author: Andrew Gierth and Atri Sharma, with contributions from Andres Freund Reviewed-By: Andres Freund, Noah Misch, Tom Lane, Svenne Krap, Tomas Vondra, Erik Rijkers, Marti Raudsepp, Pavel Stehule Discussion: CAOeZVidmVRe2jU6aMk_5qkxnB7dfmPROzM7Ur8JPW5j8Y5X-Lw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/f3d3118532175541a9a96ed78881a3b04a057128

Fujii Masao a poussé :

Simon Riggs a poussé :

Heikki Linnakangas a poussé :

  • Allow GiST distance function to return merely a lower-bound. The distance function can now set *recheck = false, like index quals. The executor will then re-check the ORDER BY expressions, and use a queue to reorder the results on the fly. This makes it possible to do kNN-searches on polygons and circles, which don't store the exact
  • Fix datatype confusion with the new lossy GiST distance functions. We can only support a lossy distance function when the distance function's datatype is comparable with the original ordering operator's datatype. The distance function always returns a float8, so we are limited to float8, and float4 (by a hard-coded cast of the float8 to float4). In light of this limitation, it seems like a good idea to have a separate 'recheck' flag for the ORDER BY expressions, so that if you have a non-lossy distance function, it still works with lossy quals. There are cases like that with the build-in or contrib opclasses, but it's plausible. There was a hidden assumption that the ORDER BY values returned by GiST match the original ordering operator's return type, but there are plenty of examples where that's not true, e.g. in btree_gist and pg_trgm. As long as the distance function is not lossy, we can tolerate that and just not return the distance to the executor (or rather, always return NULL). The executor doesn't need the distances if there are no lossy results. There was another little bug: the recheck variable was not initialized before calling the distance function. That revealed the bigger issue, as the executor tried to reorder tuples that didn't need reordering, and that failed because of the datatype mismatch. http://git.postgresql.org/pg/commitdiff/98edd617f3b62a02cb2df9b418fcc4ece45c7ec0
  • Fix docs build. Oops. http://git.postgresql.org/pg/commitdiff/8b0f105d2d179dc1085b16f0594c8fa78d13267d
  • Add archive_mode='always' option. In 'always' mode, the standby independently archives all files it receives from the primary. Original patch by Fujii Masao, docs and review by me. http://git.postgresql.org/pg/commitdiff/ffd37740ee6fcd434416ec0c5461f7040e0a11de
  • Silence another create_index regression test failure. More platform differences in the less-significant digits in output. Per buildfarm member rover_firefly, still. http://git.postgresql.org/pg/commitdiff/11a83bbedd73800db70f6f2af5a8eb10d15d39d7
  • Silence create_index regression test failure. The expected output contained some floating point values which might get rounded slightly differently on different platforms. The exact output isn't very interesting in this test, so just round it. Per buildfarm member rover_firefly. http://git.postgresql.org/pg/commitdiff/9feaba28e27820f91d13c3de6581bb3b8c3234c6
  • value in the index, but just a bounding box. Alexander Korotkov and me http://git.postgresql.org/pg/commitdiff/35fcb1b3d038a501f3f4c87c05630095abaaadab

Magnus Hagander a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Andres Freund sent in another revision of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Michael Paquier sent in a patch to improve regression tests to check LOCK TABLE and table permissions.
  • Michael Paquier sent in a patch to remove leftovers after dcae5fac (improve speed of make check-world) in git tree with check-world.
  • Abhijit Menon-Sen sent in two more revisions of a patch to create a fast bloat measurement tool.
  • Heikki Linnakangas sent in two more revisions of a patch to fix some interactions between streaming replication and WAL archiving.
  • Kaigai Kouhei sent in three more revisions of a patch to improve the custom/foreign join API.
  • Stephen Frost sent in two more revisions of a patch to add default roles for functionality.
  • Álvaro Herrera and Petr Jelinek traded patches to add an access method for sequences.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add multi-column statistics.
  • Petr A. Korobeinikov sent in another revision of a patch to support for \ev viewname and \sv viewname in psql.
  • Fabien COELHO sent in a patch to allow backslash-continuations in custom scripts.
  • Etsuro Fujita sent in a doc patch for ALTER FOREIGN TABLE.
  • Robert Haas sent in a patch to fix the documentation of BackgroundWorkerInitializeConnection.
  • Beena Emerson sent in another revision of a patch to add support for N synchronous standby servers.
  • Shigeru HANADA sent in another revision of a patch to add postgres_fdw join pushdown.

par N Bougain le lundi 18 mai 2015 à 21h40

Adrien Nayrat

Partition xlogs pleine

Cette semaine j’ai été confronté à un incident sur une instance postgres. La partition xlogs était pleine :

PANIC: could not write to file "pg_xlog/xlogtemp.12352": No space left on device
LOG: startup process (PID 12352) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure

Cette instance est répliquée sur un autre serveur et archive ses journaux de transaction sur un troisième serveur. Ce dernier était plein, Postgres est intelligent et a conservé ses journaux de transaction en attendant de pouvoir les archiver… Jusqu’à ce que la partition des journaux se retrouve pleine entraînant l’arrêt de Postgres.

Le nettoyage sur le serveur d’archivage n’était pas suffisant, il fallait libérer de la place sur la partition. La solution qui vient en tête est d’agrandir la partition. Je ne souhaitais pas agrandir la partition. J’ai donc fait un truc tout simple : Déplacer un fichier journal et faire un lien symbolique vers celui-ci. Postgres est reparti et j’ai lancé un “CHECKPOINT” immédiatement ce qui a libéré de la place. Ensuite il faut penser à supprimer le fichier car Postgres a juste supprimé le lien.
C’est plus rapide à mettre en place que d’agrandir la partition.

En fouillant sur google je suis tombé sur ce site :
Solving pg_xlog out of disk space problem on Postgres

L’auteur nous donne une astuce toute simple : Créer un fichier plein de vide. Si un jour la partition est pleine il suffit de le supprimer pour faire repartir l’instance :

 dd if=/dev/zero of=DO_NOT_MOVE_THIS_FILE bs=1MB count=100

Bilan : Il faut bien penser à superviser l’archivage de Postgres!

FacebookTwitterGoogle+ViadeoPrintEmailGoogle GmailLinkedInPartager

par Adrien Nayrat le lundi 18 mai 2015 à 20h00

dimanche 17 mai 2015

ulhume

Rooter le LG G Pad 7.0

Lorsque l&aposon achète un device Android, se pose toujours le même problème &aposcomment vais-je rooter ce machin là&apos.

Voici donc comment faire pour la tablette LG GPad 7.0.

Pourquoi faire ?

Comme mon sujet pour l&aposannée 2015 est clairement le développement d&aposapplications mobile (allez zouh, un peu de pub ;-), après m&aposêtre commis à acheter un Mac Mini et un iPad Mini, j&aposai aussi dut faire l&aposacquisition d&aposun fairphone (pour me racheter un peu...) et d&aposune tablette Android, le LG G Pad 7.0 (le Windows Phone... m&aposa été gracieusement donné, ouf !! ;-).

Concernant les deux devices Android, seul le fairphone a le bon goût d&aposêtre rooté en sortie d&aposusine. En revanche, le LG, c&aposest prison dorée... Or pour développer sous Android, rooter est l&aposétape nécessaire. Car outre l&aposaspect philosophique (je ne supporte pas l&aposidée qu&aposon m&aposempêche de faire ce que je veux de ce que j&aposachète), l&aposaccès root me permet d&aposutiliser ADB en mode réseau (c&aposest à dire sans cable USB). Et ça, je ne sais pas m&aposen passer...

Bref, trêve de blabla, voyons comment casser la bête...

Passer en mode développeur

La première étape consiste déjà à activer le mode développeur, ce qui n&aposest pas aussi évident que l&aposon pourrait se l&aposimaginer. Pour cela vous devrez aller dans les réglages, puis dans le menu à propos de la tablette, et enfin dans information sur le logiciel. Arrivé là, accrochez-vous bien, vous devez taper plusieurs fois sur le Numéro de build.

La tablette vous demande alors si vous êtes certain de votre action (non, non, j&aposai tapé 20 fois là dessus par pur hasard...), puis fera apparaître, à la positive, le menu tant utile pour les développeurs.

Dans ce menu, vous allez devoir vous rendre dans la section debuggage pour activer le debuggage de la connection USB. Ce qui est le pré-requis indispensable pour pouvoir utiliser ADB mais aussi pour pouvoir rooter l&aposappareil.

Cas n°1, KitKat

Récupération de Purple Drake

Pour ceux qui ont acquis leur tablette à sa sortie, la version Android de base est KitKat. Pour rooter cette version, le sésame s&aposappelle Purple Drake. Purple Drake Pour le rooting à proprement parlé, le sésame s&aposappelle Purple Drake. Téléchargez donc la dernière version (R3 dans mon cas) et décompressez là en local.

Pour ceux qui utilisent (encore) Wheezy

Pour ceux qui sont sur une version récente du kernel, cette étape peut être zappée. Dans mon cas, la version de GLibC incluse dans Debian Wheezy ne me permet de lancer le script tel quel avec les versions binaires d&aposadb incluse dans l&aposarchive.

Cela se règle cependant simplement en installant la version debian d&aposadb par sudo apt-get install android-adb. Ceci fait, allez dans le dossier assets de l&aposarchive décompressée de Purple Drake et éditez le fichier purpledrake_main.sh pour remplacer en ligne 16, $1 par /usr/bin.

Lorsque tout est en ordre, il ne reste plus qu&aposà lancer l&aposoutil. Assurez-vous que la tablette n&aposa aucune application de lancée (rebootez là si nécessaire), que le cable USB est bien connecté, et que dans la barre de status de la tablette vous voyez bien que la connection USB se fait en mode debuggage.

root !

Si tout est OK, lancez le rooting par sudo ./purpledrake_linux.sh. La raison du sudo ici est que sous Debian, en standard, seul root a accès au device USB. Cela peut se configurer au niveau d&aposudev mais ça me saoule un peu d&aposavoir à faire cela à chaque fois, sur chaque machine et pour chaque device. Et c&aposest d&aposailleurs l&aposune des raisons qui me fait passer ADB en mode réseau.

Une fois l&aposoutil lancé il suffit de se laisser guider par le script. Il va d&aposabord rebooter le device, puis installer un root temporaire, puis l&aposutiliser, si vous le désirez, pour mettre en place le root permanent.

Lorsque la tablette a redémarré, tout application qui demande l&aposaccès root devrait ainsi l&aposobtenir. C&aposest une première étape mais ce n&aposest pas très sécurisé, loin de là.

Super pouvoirs à la demande

Pour aller un cran plus loin, rendez vous sur le market et téléchargez SuperSU. Cette application au démarrage va remplacer la commande su fournie par Purple Drake, par une version qui vous demandera si telle application a bien le droit d&aposobtenir l&aposaccès root.

Au premier lancement, SuperSU va vous proposer une procédure de remplacement de la commande en mode normal ou recovery. J&aposai personnellement pris l&aposoption normal.

Et après une installation avec succès suivi d&aposun redémarrage de l&aposengin, tout était opérationnel.

Félicitation, votre device vous appartient !

Cas n°2, Lollipop

Si vous avez acheté votre LG plus récemment, ou si, comme moi, vous avez clické sur "mettre à jour" sans faire tourner votre cerveau, vous n&aposêtes pas sous Kitkat mais sous Lollipop. Et là, le purple drake il marche plus du tout !

Alors heureusement il y a une autre méthode qui fonctionne parfaitement mais elle nécessite... windows. Je sais, c&aposest pas cool, mais je n&aposai pas trouvé mieux.

Pour rooter, petite liste de courses :

  1. Les pilotes Android de chez LG. Perso, je les ai trouvés .
  2. Une petite application toute mignonette qui va vous simplifier le rootage, et que vous trouverez ici

Première étape, installer les pilotes sur le windows. Je ne sais pas pour vous mais chez moi cela a pris des plombes !!!

Ceci fait, connectez votre LG par le câble USB et lancez l&aposapplication puis clickez sur ROOT DEVICE. Cela va lancer une console et le script associé va redémarrer le device. Si après ce redémarrage vous vous retrouvez à nouveau sous votre home Android, c&aposest que quelque chose n&aposaura pas marché. Dans ce cas le script va (sans doute) vous dire (je traduis ;-) "désolé, mais j&aposai échoué salement à passer le device en mode série, fait le toi-même, moi j&aposattends".

En effet, le script a besoin que le device soit dans un mode très spécial de mise à jour du firmeware qui place le port USB en mode "port série". Dans ce mode, il lui sera possible de lire un numéro de série qui permettra par la suite de rooter l&aposengin.

Pour faire cela manuellement procédez comme suit sans toucher au script côté windows qui doit afficher un waiting device. Qu&aposil attende donc...

  1. Débranchez le cable,
  2. Éteignez le device par une longue pression sur le bouton d&aposallumage, puis Éteindre,
  3. Pressez le bouton qui monte le volume (c&aposest celui qui est prêt du bouton d&aposallumage),
  4. Tout en maintenant ce bouton pressé, rebranchez le cable USB
  5. Là le device doit s&aposallumer et après un temps il va se stabiliser dans un mode étrange avec une sorte de barre de progression pour la mise à jour du firmware.
  6. Le script côté windows doit logiquement se réveiller et recommencer à vous causer,
  7. Lorsque le script a terminé son boulot, le device a redémarré en mode Android classique.
  8. Le script vous demande (en anglais) de l&aposarrêter (???) en pressant les touches Control+C , puis de pressez N puis Enter.
  9. Ceci fait, vous êtes de retours sur l&aposinterface graphique.

À ce stage, normalement c&aposest tout bon et le LG est rooté. Pour tester, déverrouillez la home et sur l&aposinterface côté Windows pressez le bouton "ADB SHELL". Vous devriez voir apparaître une console avec quelque chose comme &aposshell@e7wifi:/ $&apos. Maintenant tapez su puis validez. Et là, côté Android, une boite de dialogue doit apparaître pour vous demander si vous authoriser ce passage en mode root. Acceptez. De retour sous Windows, dans la console, devrait alors s&aposafficher root@e7wifi:/ #.

C&aposest bon, vous êtes root !

Tester le tout

Pour tester, le plus simple est d&aposinstaller ADB Wifi à partir du market et de le lancer. Si les étapes précédentes ont fonctionnées, lorsque vous activerez la ADB en mode réseau, une boite de dialogue doit apparaître pour valider l&aposaccès.

Ceci fait, ADB Wifi doit vous indiquer qu&aposil est en écoute et vous fournis l&aposIP de connection. Vous n&aposavez alors plus qu&aposà vous connecter de votre machine comme ceci :

$sudo adb connect 192.168.154.21
connected to 192.168.154.21:5555
$sudo adb shell

Conclusion

Il est toujours navrant d&aposavoir à perdre du temps sur quelque chose d&aposaussi trivial.

Espérons que peu à peu les constructeurs s&aposinspire de FairPhone et arrêtent ainsi de prendre leurs clients pour des crétins... On peut toujours rêver ;-)

Please enable JavaScript to view the comments powered by Disqus.

dimanche 17 mai 2015 à 22h13

lundi 11 mai 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 10 mai 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Second try at fixing warnings caused by commit 9b43d73b3f9bef27. Commit ef3f9e642d2b2bba suppressed one cause of warnings here, but recent clang on OS X is still unhappy because we're passing a "long" to abs(). The fact that tm_gmtoff is declared as long is no doubt a hangover from days when int might be only 16 bits; but Postgres has never been able to run on such machines, so we can just cast it to int with no worries. For consistency, also cast to int in the other uses of tm_gmtoff in this stanza. Note: this code is still broken on machines that don't follow C99 integer-division-truncates-towards-zero rules. Given the lack of complaints about it, I don't feel a large desire to complicate things enough to cope with the pre-C99 rules. http://git.postgresql.org/pg/commitdiff/c90b85e4d9e4ae3bc26459cc54697e1adaa4315f
  • Improve procost estimates for some text search functions. The text search functions that involve parsing raw text into lexemes are remarkably CPU-intensive, so estimating them at the same cost as most other built-in functions seems like a mistake; moreover, doing so turns out to discourage the optimizer from using functional indexes on these functions. After some debate, we've agreed to raise procost from 1 to 100 for to_tsvector(), plainto_tsvector(), to_tsquery(), ts_headline(), ts_match_tt(), and ts_match_tq(), which are all the text search functions that parse raw text. Also increase procost for the 2-argument form of ts_rewrite() (tsquery_rewrite_query); while this function doesn't do text parsing, it does execute a user-supplied SQL query, so its previous procost of 1 is clearly a drastic underestimate. It seems reasonable to assign it the same cost we assign to PL functions by default, so 100 is the number here too. I did not bother bumping catversion for this change, since it does not break catalog compatibility with the server executable nor result in any regression test changes. Per complaint from Andrew Gierth and subsequent discussion. http://git.postgresql.org/pg/commitdiff/2503982be4ca48f48d2bb6e1d46160b23e4bb268
  • Fix incorrect declaration of citext's regexp_matches() functions. These functions should return SETOF TEXT[], like the core functions they are wrappers for; but they were incorrectly declared as returning just TEXT[]. This mistake had two results: first, if there was no match you got a scalar null result, whereas what you should get is an empty set (zero rows). Second, the 'g' flag was effectively ignored, since you would get only one result array even if there were multiple matches, as reported by Jeff Certain. While ignoring 'g' is a clear bug, the behavior for no matches might well have been thought to be the intended behavior by people who hadn't compared it carefully to the core regexp_matches() functions. So we should tread carefully about introducing this change in the back branches. Still, it clearly is a bug and so providing some fix is desirable. After discussion, the conclusion was to introduce the change in a 1.1 version of the citext extension (as we would need to do anyway); 1.0 still contains the incorrect behavior. 1.1 is the default and only available version in HEAD, but it is optional in the back branches, where 1.0 remains the default version. People wishing to adopt the fix in back branches will need to explicitly do ALTER EXTENSION citext UPDATE TO '1.1'. (I also provided a downgrade script in the back branches, so people could go back to 1.0 if necessary.) This should be called out as an incompatible change in the 9.5 release notes, although we'll also document it in the next set of back-branch release notes. The notes should mention that any views or rules that use citext's regexp_matches() functions will need to be dropped before upgrading to 1.1, and then recreated again afterwards. Back-patch to 9.1. The bug goes all the way back to citext's introduction in 8.4, but pre-9.1 there is no extension mechanism with which to manage the change. Given the lack of previous complaints it seems unnecessary to change this behavior in 9.0, anyway. http://git.postgresql.org/pg/commitdiff/b22527f29dba6395a9e950fc655d34914c960f89
  • citext's regexp_matches() functions weren't documented, either. http://git.postgresql.org/pg/commitdiff/929ca96584bef1cc7d09a8e57d26d8c3f25a92a4
  • Add missing "static" marker. Per buildfarm member pademelon. http://git.postgresql.org/pg/commitdiff/c594c750789fd98a19dcdf974b87ba9833995cf5
  • Code review for foreign/custom join pushdown patch. Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design decisions that seem pretty questionable to me, and there was quite a lot of stuff not to like about the documentation and comments. Clean up as follows: * Consider foreign joins only between foreign tables on the same server, rather than between any two foreign tables with the same underlying FDW handler function. In most if not all cases, the FDW would simply have had to apply the same-server restriction itself (far more expensively, both for lack of caching and because it would be repeated for each combination of input sub-joins), or else risk nasty bugs. Anyone who's really intent on doing something outside this restriction can always use the set_join_pathlist_hook. * Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist to better reflect what they're for, and allow these custom scan tlists to be used even for base relations. * Change make_foreignscan() API to include passing the fdw_scan_tlist value, since the FDW is required to set that. Backwards compatibility doesn't seem like an adequate reason to expect FDWs to set it in some ad-hoc extra step, and anyway existing FDWs can just pass NIL. * Change the API of path-generating subroutines of add_paths_to_joinrel, and in particular that of GetForeignJoinPaths and set_join_pathlist_hook, so that various less-used parameters are passed in a struct rather than as separate parameter-list entries. The objective here is to reduce the probability that future additions to those parameter lists will result in source-level API breaks for users of these hooks. It's possible that this is even a small win for the core code, since most CPU architectures can't pass more than half a dozen parameters efficiently anyway. I kept root, joinrel, outerrel, innerrel, and jointype as separate parameters to reduce code churn in joinpath.c --- in particular, putting jointype into the struct would have been problematic because of the subroutines' habit of changing their local copies of that variable. * Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all right for it to know about IndexOnlyScan, but if the list is to grow we should refactor the knowledge out to the callers. * Restore nodeForeignscan.c's previous use of the relcache to avoid extra GetFdwRoutine lookups for base-relation scans. * Lots of cleanup of documentation and missed comments. Re-order some code additions into more logical places. http://git.postgresql.org/pg/commitdiff/1a8a4e5cde2b7755e11bde2ea7897bd650622d3e

Andrew Dunstan a poussé :

Heikki Linnakangas a poussé :

  • Fix the same-rel optimization when creating WAL records. prev_regbuf was never set, and therefore the same-rel flag was never set on WAL records. Report and fix by Zhanq Zq http://git.postgresql.org/pg/commitdiff/ec3d976bce7e322c29f1007d19b63b7a3a1a6ee4
  • At promotion, archive last segment from old timeline with .partial suffix. Previously, we would archive the possible-incomplete WAL segment with its normal filename, but that causes trouble if the server owning that timeline is still running, and tries to archive the same segment later. It's not nice for the standby to trip up the master's archival like that. And it's pretty confusing, anyway, to have an incomplete segment in the archive that's indistinguishable from a normal, complete segment. To avoid such confusion, add a .partial suffix to the file. Or to be more precise, make a copy of the old segment under the .partial suffix, and archive that instead of the original file. pg_receivexlog also uses the .partial suffix for the same purpose, to tell apart incompletely streamed files from complete ones. There is no automatic mechanism to use the .partial files at recovery, so they will go unused, unless the administrator manually copies to them to the pg_xlog directory (and removes the .partial suffix). Recovery won't normally need the WAL - when recovering to the new timeline, it will find the same WAL on the first segment on the new timeline instead - but it nevertheless feels better to archive the file with the .partial suffix, for debugging purposes if nothing else. http://git.postgresql.org/pg/commitdiff/de7688442f5aaa03da60416a6aa3474738718803
  • Add macros to check if a filename is a WAL segment or other such file. We had many instances of the strlen + strspn combination to check for that. This makes the code a bit easier to read. http://git.postgresql.org/pg/commitdiff/179cdd098196338880bdbb39c39a788abdad4dd8

Robert Haas a poussé :

  • Use outerPlanState macro instead of referring to leffttree. This makes the executor code more consistent. It also removes an apparently superfluous NULL test in nodeGroup.c. Qingqing Zhou, reviewed by Tom Lane, and further revised by me. http://git.postgresql.org/pg/commitdiff/40f42d2a34329b0b71a1287d6fd2554298dbb713
  • Recursively fsync() the data directory after a crash. Otherwise, if there's another crash, some writes from after the first crash might make it to disk while writes from before the crash fail to make it to disk. This could lead to data corruption. Back-patch to all supported versions. Abhijit Menon-Sen, reviewed by Andres Freund and slightly revised by me. http://git.postgresql.org/pg/commitdiff/2ce439f3379aed857517c8ce207485655000fc8e
  • Fix some problems with patch to fsync the data directory. pg_win32_is_junction() was a typo for pgwin32_is_junction(). open() was used not only in a two-argument form, which breaks on Windows, but also where BasicOpenFile() should have been used. Per reports from Andrew Dunstan and David Rowley. http://git.postgresql.org/pg/commitdiff/456ff0863851d70dce679ca3f631392589e31a33
  • Avoid using a C++ keyword as a structure member name. Per request from Peter Eisentraut. http://git.postgresql.org/pg/commitdiff/1998261034ec7a948bb9b25b7cb88d014d371da1
  • Fix incorrect math in DetermineSafeOldestOffset. The old formula didn't have enough parentheses, so it would do the wrong thing, and it used / rather than % to find a remainder. The effect of these oversights is that the stop point chosen by the logic introduced in commit b69bf30b9bfacafc733a9ba77c9587cf54d06c0c might be rather meaningless. Thomas Munro, reviewed by Kevin Grittner, with a whitespace tweak by me. http://git.postgresql.org/pg/commitdiff/7be47c56af3d3013955c91c2877c08f2a0e3e6a2
  • Teach autovacuum about multixact member wraparound. The logic introduced in commit b69bf30b9bfacafc733a9ba77c9587cf54d06c0c and repaired in commits 669c7d20e6374850593cb430d332e11a3992bbcf and 7be47c56af3d3013955c91c2877c08f2a0e3e6a2 helps to ensure that we don't overwrite old multixact member information while it is still needed, but a user who creates many large multixacts can still exhaust the member space (and thus start getting errors) while autovacuum stands idly by. To fix this, progressively ramp down the effective value (but not the actual contents) of autovacuum_multixact_freeze_max_age as member space utilization increases. This makes autovacuum more aggressive and also reduces the threshold for a manual VACUUM to perform a full-table scan. This patch leaves unsolved the problem of ensuring that emergency autovacuums are triggered even when autovacuum=off. We'll need to fix that via a separate patch. Thomas Munro and Robert Haas http://git.postgresql.org/pg/commitdiff/53bb309d2d5a9432d2602c93ed18e58bd2924e15

Peter Eisentraut a poussé :

Álvaro Herrera a poussé :

  • Add geometry/range functions to support BRIN inclusion. This commit adds the following functions: box(point) -> box, bound_box(box, box) -> box, inet_same_family(inet, inet) -> bool, inet_merge(inet, inet) -> cidr, range_merge(anyrange, anyrange) -> anyrange. The first of these is also used to implement a new assignment cast from point to box. These functions are the first part of a base to implement an "inclusion" operator class for BRIN, for multidimensional data types. Author: Emre Hasegeli. Reviewed by: Andreas Karlsson http://git.postgresql.org/pg/commitdiff/3b6db1f445e14bd189ebc99ce1e5535a1c624613
  • Improve BRIN infra, minmax opclass and regression test. The minmax opclass was using the wrong support functions when cross-datatypes queries were run. Instead of trying to fix the pg_amproc definitions (which apparently is not possible), use the already correct pg_amop entries instead. This requires jumping through more hoops (read: extra syscache lookups) to obtain the underlying functions to execute, but it is necessary for correctness. Author: Emre Hasegeli, tweaked by Álvaro Review: Andreas Karlsson Also change BrinOpcInfo to record each stored type's typecache entry instead of just the OID. Turns out that the full type cache is necessary in brin_deform_tuple: the original code used the indexed type's byval and typlen properties to extract the stored tuple, which is correct in Minmax; but in other implementations that want to store something different, that's wrong. The realization that this is a bug comes from Emre also, but I did not use his patch. I also adopted Emre's regression test code (with smallish changes), which is more complete. http://git.postgresql.org/pg/commitdiff/db5f98ab4fa44bc563ec62d7b1aada4fc276d9b2

Magnus Hagander a poussé :

Bruce Momjian a poussé :

Andres Freund a poussé :

  • Represent columns requiring insert and update privileges indentently. Previously, relation range table entries used a single Bitmapset field representing which columns required either UPDATE or INSERT privileges, despite the fact that INSERT and UPDATE privileges are separately cataloged, and may be independently held. As statements so far required either insert or update privileges but never both, that was sufficient. The required permission could be inferred from the top level statement run. The upcoming INSERT ... ON CONFLICT UPDATE feature needs to independently check for both privileges in one statement though, so that is not sufficient anymore. Bumps catversion as stored rules change. Author: Peter Geoghegan. Reviewed-By: Andres Freund. http://git.postgresql.org/pg/commitdiff/2c8f4836db058d0715bc30a30655d646287ba509
  • Remove dependency on ordering in logical decoding upsert test. Buildfarm member magpie sorted the output differently than intended by Peter. "Resolve" the problem by simply not aggregating, it's not that many lines. http://git.postgresql.org/pg/commitdiff/581f4f9657fc3ab08199d02c0a4ea89c658882a6
  • Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE. The newly added ON CONFLICT clause allows to specify an alternative to raising a unique or exclusion constraint violation error when inserting. ON CONFLICT refers to constraints that can either be specified using a inference clause (by specifying the columns of a unique constraint) or by naming a unique or exclusion constraint. DO NOTHING avoids the constraint violation, without touching the pre-existing row. DO UPDATE SET ... [WHERE ...] updates the pre-existing tuple, and has access to both the tuple proposed for insertion and the existing tuple; the optional WHERE clause can be used to prevent an update from being executed. The UPDATE SET and WHERE clauses have access to the tuple proposed for insertion using the "magic" EXCLUDED alias, and to the pre-existing tuple using the table name or its alias. This feature is often referred to as upsert. This is implemented using a new infrastructure called "speculative insertion". It is an optimistic variant of regular insertion that first does a pre-check for existing tuples and then attempts an insert. If a violating tuple was inserted concurrently, the speculatively inserted tuple is deleted and a new attempt is made. If the pre-check finds a matching tuple the alternative DO NOTHING or DO UPDATE action is taken. If the insertion succeeds without detecting a conflict, the tuple is deemed inserted. To handle the possible ambiguity between the excluded alias and a table named excluded, and for convenience with long relation names, INSERT INTO now can alias its target table. Bumps catversion as stored rules change. Author: Peter Geoghegan, with significant contributions from Heikki Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes. Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs, Dean Rasheed, Stephen Frost and many others. http://git.postgresql.org/pg/commitdiff/168d5805e4c08bed7b95d351bf097cff7c07dd65
  • Minor ON CONFLICT related comments and doc fixes. Geoff Winkless, Stephen Frost, Peter Geoghegan and me. http://git.postgresql.org/pg/commitdiff/e8898e9169c851c2b8c98f981c1c4755a5758f8e
  • Fix two problems in infer_arbiter_indexes(). The first is a pretty simple bug where a relcache entry is used after the relation is closed. In this particular situation it does not appear to have bad consequences unless compiled with RELCACHE_FORCE_RELEASE. The second is that infer_arbiter_indexes() skipped indexes that aren't yet valid according to indcheckxmin. That's not required here, because uniqueness checks don't care about visibility according to an older snapshot. While thats not really a bug, it makes things undesirably non-deterministic. There is some hope that this explains a test failure on buildfarm member jaguarundi. Discussion: 9096.1431102730@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/bab64ef9e8bc56fa5db9bd41cefb54c3d8051dbe

Stephen Frost a poussé :

  • Remove reference to src/tools/backend/index.html. src/tools/backend was removed back in 63f1ccd, but backend/storage/lmgr/README didn't get the memo. Author: Amit Langote http://git.postgresql.org/pg/commitdiff/195fbd40123b85ba8a44ca273b17d699e30ec6a8
  • Add pg_file_settings view and function. The function and view added here provide a way to look at all settings in postgresql.conf, any #include'd files, and postgresql.auto.conf (which is what backs the ALTER SYSTEM command). The information returned includes the configuration file name, line number in that file, sequence number indicating when the parameter is loaded (useful to see if it is later masked by another definition of the same parameter), parameter name, and what it is set to at that point. This information is updated on reload of the server. This is unfiltered, privileged, information and therefore access is restricted to superusers through the GRANT system. Author: Sawada Masahiko, various improvements by me. Reviewers: David Steele http://git.postgresql.org/pg/commitdiff/a97e0c3354ace5d74c6873cd5e98444757590be8
  • Bump catversion for pg_file_settings. Pointed out by Andres (thanks!) Apologies for not including it in the initial patch. http://git.postgresql.org/pg/commitdiff/4b342fb591ebb556cab18fc999c8710e9733e5bb
  • Modify pg_stat_get_activity to build a tuplestore. This updates pg_stat_get_activity() to build a tuplestore for its results instead of using the old-style multiple-call method. This simplifies the function, though that wasn't the primary motivation for the change, which is that we may turn it into a helper function which can filter the results (or not) much more easily. http://git.postgresql.org/pg/commitdiff/f91feba8776eb66008cdb73b3a8c0c7c08cc54d9
  • Change default for include_realm to 1. The default behavior for GSS and SSPI authentication methods has long been to strip the realm off of the principal, however, this is not a secure approach in multi-realm environments and the use-case for the parameter at all has been superseded by the regex-based mapping support available in pg_ident.conf. Change the default for include_realm to be '1', meaning that we do NOT remove the realm from the principal by default. Any installations which depend on the existing behavior will need to update their configurations (ideally by leaving include_realm set to 1 and adding a mapping in pg_ident.conf, but alternatively by explicitly setting include_realm=0 prior to upgrading). Note that the mapping capability exists in all currently supported versions of PostgreSQL and so this change can be done today. Barring that, existing users can update their configurations today to explicitly set include_realm=0 to ensure that the prior behavior is maintained when they upgrade. This needs to be noted in the release notes. Per discussion with Magnus and Peter. http://git.postgresql.org/pg/commitdiff/9a0884176fdfa51551d6a3b26fa0e1b216c3e4c2
  • Improve ParseConfigFp comment wrt head/tail. The head_p and tail_p pointers passed to ParseConfigFp() are actually input/output parameters, not strictly output paramaters. This updates the function comment to reflect that. Per discussion with Tom. http://git.postgresql.org/pg/commitdiff/0cf56f14dd15532fec930b502cb6457023b01ef8
  • Correct reindexdb documentation. --schema takes a schema, not a table. Author: Sawada Masahiko http://git.postgresql.org/pg/commitdiff/f0a4b20bb9f91bdc0d60ff8732ee0195b0dfdd73

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in a patch to fix a potential pointer dereference in plperl.c caused by transforms patch.
  • Petr Korobeinikov sent in a patch to add \ev and \sv to edit and show the creation scripts for views, respectively, to psql.
  • Álvaro Herrera sent in two more revisions of a patch to add deparsing utilities.
  • Arjen Nienhuis sent in a patch to have GB18030 handle more than 2-byte Unicode code points.
  • Fabien COELHO sent in two more revisions of a patch to extend pgbench expressions with functions.
  • Emre Hasegeli and Álvaro Herrera traded patches to add a BRIN range operator class.
  • Tom Lane sent in another revision of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Tomas Vondra sent in another revision of a patch to add multivariate statistics.
  • Fabien COELHO sent in a patch to pgbench to allow '=' in \set.
  • SAWADA Masahiko and Fabrízio de Royes Mello traded patches to add REINDEX ... VERBOSE.
  • Stephen Frost sent in two more revisions of a patch to add default roles for sets of administrative functions.
  • Fabien COELHO sent in another revision of a patch to remove nclients/nthreads constraint from pgbench.
  • Stas Kelvich sent in another revision of a patch to add kNN support for the cube extension.
  • Michael Paquier sent in a patch to redefine subxcnt as uint32 for consistency with xcnt.
  • Michael Paquier sent in a patch to make more precise the rounding behavior of numeric and double precision in docs.
  • Kaigai Kouhei sent in three more revisions of a patch to add a custom/foreign join API.
  • Kaigai Kouhei sent in a patch to fix regtest policy for sepgsql.
  • Tom Lane sent in a patch to fix a bug in HOT.

par N Bougain le lundi 11 mai 2015 à 00h49

Nouvelles hebdomadaires de PostgreSQL - 3 mai 2015

La classe Consistent State's PG Admin aura lieu à Denver du 18 au 21 mai. Les inscriptions sont ouvertes jusqu'au 14 : http://www.consistentstate.com/store/c1/Featured_Products.html

PGConf Silicon Valley 2015 se tiendra au centre de convention sud de San Francisco les 17 & 18 novembre. L'appel à conférenciers porte jusqu'au 15 juin : http://www.pgconfsv.com

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Andrew Dunstan a poussé :

  • fix escaping of brackets for m4 broken in b6b2149e48aa61981ae0199c963d5145a37c258c http://git.postgresql.org/pg/commitdiff/f707b53449f3ab6998c615b746ad01f5775723a3
  • Enable transforms modules to build and run with Mingw builds. These modules were all missing essential Windows scaffolding, including resources files and descriptions, and links to the relevant library import files. This latter item means that the modules can't be built with pgxs on Windows, as we don't install the import files. If we ever decide to install them this restriction could probably be removed. Also, as with plperl we need to make sure that perl's CORE directory is last on the include list, as on Windows it appears to contain some headers with names that clash with names of some headers we include. http://git.postgresql.org/pg/commitdiff/f802c6ddba143bd88512b5fc34e84ae0b4883284
  • Fix python_includespec on Windows at configure time. By converting to using forward slashes at configure time we avoid having to repeat the logic anywhere that this is needed, such as in transforms modules for plpython. http://git.postgresql.org/pg/commitdiff/b6b2149e48aa61981ae0199c963d5145a37c258c
  • Make hstore_plperl's build more like plperl's. This involves moving perl's CORE library to the end of the include list, and adding other compilation settings that plperl uses. This won't completely fix the breakage currently being seen by gcc builds on Windows, but it will let the build get further, and should be wholly benign, if not beneficial, on *nix. http://git.postgresql.org/pg/commitdiff/77477e745be534c5925cf7cb8b9c6a7698c575a3
  • Enable transforms tests for python 2 on MSVC builds. Currently regression tests for python 3 are disabled on MSVC, and these tests fail with python 3, too, so we have some work to do to enable both. Meanwhile, all the buildfarm hosts seem to be building with python 2 anyway, so this at least gets us some coverage. Original patch from Michael Paquier, significantly modified by me. http://git.postgresql.org/pg/commitdiff/eb010637dd47be65e5d8b22d6965c2e96f6937b8
  • Fix MSVC builds for contrib transforms modules. With this patch the MSVC build and installation will work correctly with the transforms. However the python transform tests for hstore and ltree are still disabled pending some further adjustments. Michael Paquier with some tweaks from me. http://git.postgresql.org/pg/commitdiff/cbf9f0ec312e54481e777ffbe941b548d909081e
  • Fix vcbuild failures and chkpass dependency caused by 854adb8. Switching the Windows build scripts to use forward slashes instead of backslashes has caused a couple of issues in VC builds: - The file tree list was not correctly generated, build script generating vcproj file missing tree dependencies when listing items in Filter. - VC builds do not accept file paths with forward slashes, perhaps it could be possible to use a Condition but it seems safer to simply enforce the file paths to use backslashes in the vcproj files. - chkpass had an unneeded dependency with libpgport and libpgcommon to make build succeed but actually it is not necessary as crypt.c is already listed for this project and should be replaced with a fake name as it is a unique file. Michael Paquier http://git.postgresql.org/pg/commitdiff/06ca28d5ab2f810ef25e718e0d71f2233542c151

Stephen Frost a poussé :

  • Improve qual pushdown for RLS and SB views. The original security barrier view implementation, on which RLS is built, prevented all non-leakproof functions from being pushed down to below the view, even when the function was not receiving any data from the view. This optimization improves on that situation by, instead of checking strictly for non-leakproof functions, it checks for Vars being passed to non-leakproof functions and allows functions which do not accept arguments or whose arguments are not from the current query level (eg: constants can be particularly useful) to be pushed down. As discussed, this does mean that a function which is pushed down might gain some idea that there are rows meeting a certain criteria based on the number of times the function is called, but this isn't a particularly new issue and the documentation in rules.sgml already addressed similar covert-channel risks. That documentation is updated to reflect that non-leakproof functions may be pushed down now, if they meet the above-described criteria. Author: Dean Rasheed, with a bit of rework to make things clearer, along with comment and documentation updates from me. http://git.postgresql.org/pg/commitdiff/dcbf5948e12aa60b4d6ab65b6445897dfc971e01

Andres Freund a poussé :

  • Use a fd opened for read/write when syncing slots during startup. Some operating systems, including the reporter's windows, return EBADFD or similar when fsync() is invoked on a O_RDONLY file descriptor. Unfortunately RestoreSlotFromDisk() does exactly that; which causes failures after restarts in at least some scenarios. If you hit the bug the error message will be something like ERROR: could not fsync file "pg_replslot/$name/state": Bad file descriptor Simply use O_RDWR instead of O_RDONLY when opening the relevant file descriptor to fix the bug. Unfortunately I have no way of verifying the fix, but we've seen similar problems in the past. This bug goes back to 9.4 where slots were introduced. Backpatch accordingly. Reported-By: Patrice Drolet Bug: #13143: Discussion: 20150424101006.2556.60897@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/dfbaed459754e71e01bb0cc90a12802bba3f9786
  • Correct replication origin's use of UINT16_MAX to PG_UINT16_MAX. We can't rely on UINT16_MAX being present, which is why we introduced PG_UINT16_MAX... Buildfarm animal bowerbird via Andrew Gierth. http://git.postgresql.org/pg/commitdiff/e0f26fc76532defd06caf79d711fa99cea83c532
  • Fix unaligned memory access in xlog parsing due to replication origin patch. ParseCommitRecord() accessed xl_xact_origin directly. But the chunks in the commit record's data only have 4 byte alignment, whereas xl_xact_origin's members require 8 byte alignment on some platforms. Update comments to make not of that and copy the record to stack local storage before reading. With help from Stefan Kaltenbrunner in pinning down the buildfarm and verifying the fix. http://git.postgresql.org/pg/commitdiff/1db12da85bee7a01abfbbf2798904347e4d9515a
  • Introduce replication progress tracking infrastructure. When implementing a replication solution ontop of logical decoding, two related problems exist: * How to safely keep track of replication progress * How to change replication behavior, based on the origin of a row; e.g. to avoid loops in bi-directional replication setups The solution to these problems, as implemented here, consist out of three parts: 1) 'replication origins', which identify nodes in a replication setup. 2) 'replication progress tracking', which remembers, for each replication origin, how far replay has progressed in a efficient and crash safe manner. 3) The ability to filter out changes performed on the behest of a replication origin during logical decoding; this allows complex replication topologies. E.g. by filtering all replayed changes out. Most of this could also be implemented in "userspace", e.g. by inserting additional rows contain origin information, but that ends up being much less efficient and more complicated. We don't want to require various replication solutions to reimplement logic for this independently. The infrastructure is intended to be generic enough to be reusable. This infrastructure also replaces the 'nodeid' infrastructure of commit timestamps. It is intended to provide all the former capabilities, except that there's only 2^16 different origins; but now they integrate with logical decoding. Additionally more functionality is accessible via SQL. Since the commit timestamp infrastructure has also been introduced in 9.5 (commit 73c986add) changing the API is not a problem. For now the number of origins for which the replication progress can be tracked simultaneously is determined by the max_replication_slots GUC. That GUC is not a perfect match to configure this, but there doesn't seem to be sufficient reason to introduce a separate new one. Bumps both catversion and wal page magic. Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer Discussion: 20150216002155.GI15326@awork2.anarazel.de, 20140923182422.GA15776@alap3.anarazel.de, 20131114172632.GE7522@alap2.anarazel.de http://git.postgresql.org/pg/commitdiff/5aa2350426c4fdb3d04568b65aadac397012bbcb
  • Copy editing of the replication origins patch. Michael Paquier and myself. http://git.postgresql.org/pg/commitdiff/2b22795b32576fa7173b501b646581a17de35902

Álvaro Herrera a poussé :

  • Protect against multixact members wraparound. Multixact member files are subject to early wraparound overflow and removal: if the average multixact size is above a certain threshold (see note below) the protections against offset overflow are not enough: during multixact truncation at checkpoint time, some pg_multixact/members files would be removed because the server considers them to be old and not needed anymore. This leads to loss of files that are critical to interpret existing tuples's Xmax values. To protect against this, since we don't have enough info in pg_control and we can't modify it in old branches, we maintain shared memory state about the oldest value that we need to keep; we use this during new multixact creation to abort if an old still-needed file would get overwritten. This value is kept up to date by checkpoints, which makes it not completely accurate but should be good enough. We start emitting warnings sometime earlier, so that the eventual multixact-shutdown doesn't take DBAs completely by surprise (more precisely: once 20 members SLRU segments are remaining before shutdown.) On troublesome average multixact size: The threshold size depends on the multixact freeze parameters. The oldest age is related to the greater of multixact_freeze_table_age and multixact_freeze_min_age: anything older than that should be removed promptly by autovacuum. If autovacuum is keeping up with multixact freezing, the troublesome multixact average size is (2^32-1) / Max(freeze table age, freeze min age) or around 28 members per multixact. Having an average multixact size larger than that will eventually cause new multixact data to overwrite the data area for older multixacts. (If autovacuum is not able to keep up, or there are errors in vacuuming, the actual maximum is multixact_freeeze_max_age instead, at which point multixact generation is stopped completely. The default value for this limit is 400 million, which means that the multixact size that would cause trouble is about 10 members). Initial bug report by Timothy Garnett, bug #12990 Backpatch to 9.3, where the problem was introduced. Authors: Álvaro Herrera, Thomas Munro Reviews: Thomas Munro, Amit Kapila, Robert Haas, Kevin Grittner http://git.postgresql.org/pg/commitdiff/b69bf30b9bfacafc733a9ba77c9587cf54d06c0c
  • Code review for multixact bugfix. Reword messages, rename a confusingly named function. Per Robert Haas. http://git.postgresql.org/pg/commitdiff/d3821e70c9b6d76083f4eb0f4cc25716e961c89d
  • Fix pg_upgrade's multixact handling (again) We need to create the pg_multixact/offsets file deleted by pg_upgrade much earlier than we originally were: it was in TrimMultiXact(), which runs after we exit recovery, but it actually needs to run earlier than the first call to SetMultiXactIdLimit (before recovery), because that routine already wants to read the first offset segment. Per pg_upgrade trouble report from Jeff Janes. While at it, silence a compiler warning about a pointless assert that an unsigned variable was being tested non-negative. This was a signed constant in Thomas Munro's patch which I changed to unsigned before commit. Pointed out by Andres Freund. http://git.postgresql.org/pg/commitdiff/669c7d20e6374850593cb430d332e11a3992bbcf
  • Fix up some loose ends for CURRENT_USER as RoleSpec. In commit 31eae6028eca4, some documents were not updated to show the new capability; fix that. Also, the error message you get when CURRENT_USER and SESSION_USER are used in a context that doesn't accept them could be clearer about it being a problem only in those contexts; so add the word "here". Author: Kyotaro HORIGUCHI His patch submission also included changes to GRANT/REVOKE, but those seemed more controversial, so I left them out. We can reconsider these changes later. http://git.postgresql.org/pg/commitdiff/9d396af46357df1243aff4a9ca4f4987e4fe6024

Tom Lane a poussé :

  • Fix ATSimpleRecursion() to allow recursion from a foreign table. This is necessary in view of the changes to allow foreign tables to be full members of inheritance hierarchies, but I (tgl) unaccountably missed it in commit cb1ca4d800621dcae67ca6c799006de99fa4f0a5. Noted by Amit Langote, patch by Etsuro Fujita http://git.postgresql.org/pg/commitdiff/ad9f08f70636051b5d5fe8d57062994b7335a960
  • Fix another test for RELKIND_RELATION that should allow foreign tables now. I thought I'd gone through all of these before, but a fresh review found this one too. (Perhaps it would be better to just delete this test and let the failure occur later, but for the moment I'll preserve the logic.) The case that this was rejecting is like CREATE FOREIGN TABLE ft (f1 int ...) ...; CREATE TABLE c1 (UNIQUE(f1)) INHERITS(ft); http://git.postgresql.org/pg/commitdiff/290713e31a1ee04eed7877985a4c28a30fd0d1db
  • Fix overlooked relcache invalidation in ALTER TABLE ... ALTER CONSTRAINT. When altering the deferredness state of a foreign key constraint, we correctly updated the catalogs and then invalidated the relcache state for the target relation ... but that's not the only relation with relevant triggers. Must invalidate the other table as well, or the state change fails to take effect promptly for operations triggered on the other table. Per bug #13224 from Christian Ullrich. In passing, reorganize regression test case for this feature so that it isn't randomly injected into the middle of an unrelated test sequence. Oversight in commit f177cbfe676dc2c7ca2b206c54d6bf819feeea8b. Back-patch to 9.4 where the faulty code was added. http://git.postgresql.org/pg/commitdiff/a4820434c1a62e0c5f4051a31ad8b4a11f0a6ad7

Bruce Momjian a poussé :

Robert Haas a poussé :

  • psql: Improve tab completion for ALTER FOREIGN TABLE. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/c6e96a2f986e4dad72c14b14d4cc17d02b2a6aad
  • Attempt to fix some compiler warnings. http://git.postgresql.org/pg/commitdiff/ef3f9e642d2b2bba8933b4cff4039ce0d354ce08
  • Remove enum-related special cases for catalog scans. When this code was written, catalog scans were normally performed using SnapshotNow, making special handling necessary here. Now, however, all catalog scans use MVCC snapshots, so we can change these cases to look more like what we do for catalog scans elsewhere in the code. Per discussion with Tom Lane and a reminder from Bruce Momjian. http://git.postgresql.org/pg/commitdiff/9b6a0ce5f060315f900de7b398d5197a2e42f2f2
  • Update .gitignore for new rmgr, changed paths. http://git.postgresql.org/pg/commitdiff/fe72c4c55bd65772aa5a4130309150ffca1749ab
  • Add <literal> markup, for consistency. This file isn't entirely consistent about whether "on" and "off" should be marked up with <literal>, but it doesn't make much sense to be inconsistent within a single sentence. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/49601ab1a6df9059f7000f78af43f2e74afefa59
  • Create an infrastructure for parallel computation in PostgreSQL. This does four basic things. First, it provides convenience routines to coordinate the startup and shutdown of parallel workers. Second, it synchronizes various pieces of state (e.g. GUCs, combo CID mappings, transaction snapshot) from the parallel group leader to the worker processes. Third, it prohibits various operations that would result in unsafe changes to that state while parallelism is active. Finally, it propagates events that would result in an ErrorResponse, NoticeResponse, or NotifyResponse message being sent to the client from the parallel workers back to the master, from which they can then be sent on to the client. Robert Haas, Amit Kapila, Noah Misch, Rushabh Lathia, Jeevan Chalke. Suggestions and review from Andres Freund, Heikki Linnakangas, Noah Misch, Simon Riggs, Euler Taveira, and Jim Nasby. http://git.postgresql.org/pg/commitdiff/924bcf4f16d54c55310b28f77686608684734f42
  • Allow FDWs and custom scan providers to replace joins with scans. Foreign data wrappers can use this capability for so-called "join pushdown"; that is, instead of executing two separate foreign scans and then joining the results locally, they can generate a path which performs the join on the remote server and then is scanned locally. This commit does not extend postgres_fdw to take advantage of this capability; it just provides the infrastructure. Custom scan providers can use this in a similar way. Previously, it was only possible for a custom scan provider to scan a single relation. Now, it can scan an entire join tree, provided of course that it knows how to produce the same results that the join would have produced if executed normally. KaiGai Kohei, reviewed by Shigeru Hanada, Ashutosh Bapat, and me. http://git.postgresql.org/pg/commitdiff/e7cb7ee14555cc9c5773e2c102efd6371f6f2005
  • Deparse named arguments to use the new => operator instead of :=. Tom Lane pointed out that this wasn't done, and asked whether that was intentional. Subsequent discussion was in favor of making the change, so here we go. http://git.postgresql.org/pg/commitdiff/e044a44949c5b9c9f548e8a1cd4bf0b50fb2a1cc

Peter Eisentraut a poussé :

  • Fix parallel make risk with new check temp-install setup. The "check" target no longer needs to depend on "all", because it now runs "install" directly, which in turn depends on "all". Doing both will cause problems with parallel make, because two builds will run next to each other. Also remove the redirection of the temp-install output into a log file. This was appropriate when this was done from within pg_regress, but now it's just a regular make run, and especially with the above changes this will now take the place of running the "all" target before the test suites. problem report by Jeff Janes, patch in part by Michael Paquier http://git.postgresql.org/pg/commitdiff/dbf2ec1a1c053379e2f9a5913979a1ca4dccbd43
  • Fix shared libpython detection on OS X. Apparently, looking for an appropriately named file doesn't work on some older versions, so put the back the explicit platform detection. http://git.postgresql.org/pg/commitdiff/010aa420b9dd5393490c8793f43949dca269a753
  • hstore_plperl: Move port-specific parts to later in the makefile. PORTNAME isn't set until the global makefiles have been included. http://git.postgresql.org/pg/commitdiff/e30a864963c5f98bafeaf3005f258ff90916da7c
  • Windows also needs an override of the shared libpython detection http://git.postgresql.org/pg/commitdiff/67df9782e94765a281a82a7672d39ebb4bf58828
  • Move interpreter shared library detection to configure. For building PL/Perl, PL/Python, and PL/Tcl, we need a shared library of libperl, libpython, and libtcl, respectively. Previously, this was checked in the makefiles, skipping the PL build with a warning if no shared library was available. Now this is checked in configure, with an error if no shared library is available. The previous situation arose because in the olden days, the configure options --with-perl, --with-python, and --with-tcl controlled whether frontend interfaces for those languages would be built. The procedural languages were added later, and shared libraries were often not available in the beginning. So it was decided skip the builds of the procedural languages in those cases. The frontend interfaces have since been removed from the tree, and shared libraries are now available most of the time, so that setup makes much less sense now. Also, the new setup allows contrib modules and pgxs users to rely on the respective PLs being available based on configure flags. http://git.postgresql.org/pg/commitdiff/d664a10f9623fd2198b257e513bce849d439a773
  • Make hstore_plperl's build even more like plperl's. Combine the two places that set CPPFLAGS into one. Also, some settings should be restricted to Windows only. More precisely, -Wno-comment is a GCC-only option, but Windows in a makefile implies GCC at the moment. Also, since -Wno-comment is more properly a preprocessor option, move it to CPPFLAGS to simplify things a bit. http://git.postgresql.org/pg/commitdiff/0fd764647a9910a340359bb319929b70317b2ae4

Magnus Hagander a poussé :

Heikki Linnakangas a poussé :

  • Fix pg_rewind regression failure after "fast promotion". pg_rewind looks at the control file to determine the server's timeline. If the standby performs a "fast promotion", the timeline ID in the control file is not updated until the next checkpoint. The startup process requests a checkpoint immediately after promotion, so this is unlikely to be an issue in the real world, but the regression suite ran pg_rewind so quickly after promotion that the checkpoint had not yet completed. Reported by Stephen Frost http://git.postgresql.org/pg/commitdiff/484a848a73fc5a76c16bc73626b290154b6a57b4

Noah Misch a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Etsuro Fujita sent in two revisions of a patch to allow postgres_fdw option to control whether check constraints on remote tables are included in the definitions of foreign tables imported from a remote PG server when performing IMPORT FOREIGN SCHEMA.
  • Fabien COELHO sent in a patch to remove nclients/nthreads constraint from pgbench.
  • Petr Jelinek sent in another revision of a patch to implement a sequence access method.
  • Stephen Frost sent in another revision of a patch to change default for include_realm to 0.
  • Fabien COELHO sent in a patch to remove the pgbench thread fork-emulation code and simplify things where possible, especially around pthread_create and pthread_join.
  • Jim Nasby sent in a patch to allow SQL/plpgsql functions to accept record.
  • Bruce Momjian sent in two more revisions of a patch to make tables created by CREATE TABLE (...LIKE...) have OIDs if any of its parents did.
  • SAWADA Masahiko sent in six more revisions of a patch to create a pg_file_settings view.
  • Pavel Stehule sent in two more revisions of a patch to control context of RAISE in PL/pgsql.
  • Shigeru HANADA sent in another revision of a patch to add foreign joins.
  • Jeff Janes sent in a patch to support contrib extensions in the upcoming multivariate statistics patch.
  • Tomas Vondra sent in a patch to teach the expression walker about RestrictInfo.
  • Fabrízio de Royes Mello sent in a patch to implement COMMENT ON CURRENT DATABASE IS...
  • Jeff Janes sent in a patch to fix a compiler warning in the recent transforms feature.
  • Alexander Korotkov sent in a patch to add selectivity estimation functions to &&, @>, <@ operators in the intarray contrib extension.
  • David Rowley sent in a patch to fix some issues that came up with the new replication identifiers.
  • Petr Jelinek sent in another revision of a patch to add mogrify and indent features for the jsonb data type.
  • John Gorman sent in a patch to rectify some incompatible trig error handling.
  • Denis Kirjanov sent in a patch to enforce access control on security labels defined by admin and prohibit users from relabeling the objects.
  • Bruce Momjian sent in a patch to pg_upgrade to quote directory names in delete_old_cluster script.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add regrole.
  • Pavel Stehule sent in a patch to fix a bug in the implementation of errhidecontext().
  • SAWADA Masahiko sent in a patch to add a bit indicating frozen-ness to the visibility map.
  • Robert Haas sent in another revision of a patch to allow making vacuum failures not fatal in pgbench.
  • SAWADA Masahiko sent in two more revisions of a patch to implement REINDEX ... VERBOSE.
  • Qingqing Zhou sent in two more revisions of a patch to use outerPlanState() consistently in the executor code.
  • David Steele sent in another revision of a patch to implement pg_audit.
  • Abhijit Menon-Sen and Robert Haas traded patches to recursively fsync PGDATA at startup after a crash.
  • Bruce Momjian and Tom Lane traded patches to add a procost for to_tsvector.
  • Emre Hasegeli sent in another revision of a patch to add a BRIN range operator class.

par N Bougain le lundi 11 mai 2015 à 00h44

vendredi 1 mai 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 26 avril 2015

Le PGDay Campinas 2015 aura lieu à Campinas (Brésil) le 7 août. L'appel à conférenciers expire le 31 mai : http://pgdaycampinas.com.br/english/

L'appel à conférenciers pour la session PostgreSQL n°7 (24 septembre 2015 à Paris) est lancé jusqu'au 15 juin 2015 : call-for-paper <AT> postgresql-sessions <DOT> org. http://www.postgresql-sessions.org/7/about

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Peter Eisentraut a poussé :

Bruce Momjian a poussé :

Andres Freund a poussé :

Robert Haas a poussé :

Heikki Linnakangas a poussé :

Stephen Frost a poussé :

  • Pull in tableoid for inheiritance with rowMarks. As noted by Etsuro Fujita [1] and Dean Rasheed[2], cb1ca4d800621dcae67ca6c799006de99fa4f0a5 changed ExecBuildAuxRowMark() to always look for the tableoid in the target list, but didn't also change preprocess_targetlist() to always include the tableoid. This resulted in errors with soon-to-be-added RLS with inheritance tests, and errors when using inheritance with foreign tables. Authors: Etsuro Fujita and Dean Rasheed (independently) Minor word-smithing on the comments by me. [1] 552CF0B6.8010006@lab.ntt.co.jp [2] CAEZATCVmFUfUOwwhnBTcgi6AquyjQ0-1fyKd0T3xBWJvn+xsFA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/4ccc5bd28e7f0c0d1b221683398ae178515b9f76
  • RLS fixes, new hooks, and new test module. In prepend_row_security_policies(), defaultDeny was always true, so if there were any hook policies, the RLS policies on the table would just get discarded. Fixed to start off with defaultDeny as false and then properly set later if we detect that only the default deny policy exists for the internal policies. The infinite recursion detection in fireRIRrules() didn't properly manage the activeRIRs list in the case of WCOs, so it would incorrectly report infinite recusion if the same relation with RLS appeared more than once in the rtable, for example "UPDATE t ... FROM t ...". Further, the RLS expansion code in fireRIRrules() was handling RLS in the main loop through the rtable, which lead to RTEs being visited twice if they contained sublink subqueries, which prepend_row_security_policies() attempted to handle by exiting early if the RTE already had securityQuals. That doesn't work, however, since if the query involved a security barrier view on top of a table with RLS, the RTE would already have securityQuals (from the view) by the time fireRIRrules() was invoked, and so the table's RLS policies would be ignored. This is fixed in fireRIRrules() by handling RLS in a separate loop at the end, after dealing with any other sublink subqueries, thus ensuring that each RTE is only visited once for RLS expansion. The inheritance planner code didn't correctly handle non-target relations with RLS, which would get turned into subqueries during planning. Thus an update of the form "UPDATE t1 ... FROM t2 ..." where t1 has inheritance and t2 has RLS quals would fail. Fix by making sure to copy in and update the securityQuals when they exist for non-target relations. process_policies() was adding WCOs to non-target relations, which is unnecessary, and could lead to a lot of wasted time in the rewriter and the planner. Fix by only adding WCO policies when working on the result relation. Also in process_policies, we should be copying the USING policies to the WITH CHECK policies on a per-policy basis, fix by moving the copying up into the per-policy loop. Lastly, as noted by Dean, we were simply adding policies returned by the hook provided to the list of quals being AND'd, meaning that they would actually restrict records returned and there was no option to have internal policies and hook-based policies work together permissively (as all internal policies currently work). Instead, explicitly add support for both permissive and restrictive policies by having a hook for each and combining the results appropriately. To ensure this is all done correctly, add a new test module (test_rls_hooks) to test the various combinations of internal, permissive, and restrictive hook policies. Largely from Dean Rasheed (thanks!): CAEZATCVmFUfUOwwhnBTcgi6AquyjQ0-1fyKd0T3xBWJvn+xsFA@mail.gmail.com Author: Dean Rasheed, though I added the new hooks and test module. http://git.postgresql.org/pg/commitdiff/0bf22e0c8b1114ae37939c500535307abefd38e1
  • Fix installcheck for test_rls_hooks. As pointed out by the buildfarm, test_rls_hooks wasn't functioning properly with a clean installcheck. test_rls_hooks needs to explicitly load the library with the hooks in it, to allow installcheck to work; using the --temp-config doesn't help since that isn't used when running installcheck and it isn't exactly fair to the buildfarm to modify the installed config prior to calling installcheck. Also, have test_rls_hooks clean up after itself. http://git.postgresql.org/pg/commitdiff/450fa1b5ba0e986a20d8c017500c0c0bbf1e0b4b
  • Copy the relation name for error reporting in WCOs. In get_row_security_policies(), we need to make a copy of the relation name when building the WithCheckOptions structure, since RelationGetRelationName just returns a pointer into the local Relation structure. The relation name in the WCO structure is only used for error reporting. Pointed out by Robert and Christian Ullrich, who noted that the buildfarm members with -DCLOBBER_CACHE_ALWAYS were failing. http://git.postgresql.org/pg/commitdiff/cb087ec03bbb1d52845a4de83a6bf634dac2639f
  • Perform RLS WITH CHECK before constraints, etc. The RLS capability is built on top of the WITH CHECK OPTION system which was added for auto-updatable views, however, unlike WCOs on views (which are mandated by the SQL spec to not fire until after all other constraints and checks are done), it makes much more sense for RLS checks to happen earlier than constraint and uniqueness checks. This patch reworks the structure which holds the WCOs a bit to be explicitly either VIEW or RLS checks and the RLS-related checks are done prior to the constraint and uniqueness checks. This also allows better error reporting as we are now reporting when a violation is due to a WITH CHECK OPTION and when it's due to an RLS policy violation, which was independently noted by Craig Ringer as being confusing. The documentation is also updated to include a paragraph about when RLS WITH CHECK handling is performed, as there have been a number of questions regarding that and the documentation was previously silent on the matter. Author: Dean Rasheed, with some kabitzing and comment changes by me. http://git.postgresql.org/pg/commitdiff/e89bd02f58ac07e44e0388a32b7ee1b42f1fd7c6
  • Fix file comment for test_rls_hooks.c. The file-level comment wasn't updated when it was copied from the shared memory queue test module. Fixed. Noted by Dean Rasheed. http://git.postgresql.org/pg/commitdiff/410cbfd6dd778e8f388fd0d7ee9d84f833700da5

Álvaro Herrera a poussé :

  • Use the right type OID after creating a shell type. Commit a2e35b53c39b2a neglected to update the type OID to use further down in DefineType when TypeShellMake was changed to return ObjectAddress instead of OID (it got it right in DefineRange, however.) This resulted in an internal error message being issued when looking up I/O functions. Author: Michael Paquier. Also add Asserts() to a couple of other places to ensure that the type OID being used is as expected. http://git.postgresql.org/pg/commitdiff/50a16e30ebd76e70fc76abb2c8f0cd1e71deac41

Tom Lane a poussé :

  • Fix obsolete comment in set_rel_size(). The cross-reference to set_append_rel_pathlist() was obsoleted by commit e2fa76d80ba571d4de8992de6386536867250474, which split what had been set_rel_pathlist() and child routines into two sets of functions. But I (tgl) evidently missed updating this comment. Back-patch to 9.2 to avoid unnecessary divergence among branches. Amit Langote http://git.postgresql.org/pg/commitdiff/70d44dd9de2b781436ef1d55906614d241e02249
  • Fix up .gitignore and cleanup actions in some src/test/ subdirectories. examples/, locale/, and thread/ lacked .gitignore files and were also not connected up to top-level "make clean" etc. This had escaped notice because none of those directories are built in normal scenarios. Still, they have working Makefiles, so if someone does a "make" in one of these directories it would be good if (a) git doesn't bleat about the product files and (b) cleaning up removes them. This is a longstanding oversight, but since this behavior is probably only of interest to developers, there seems no need for back-patching. Michael Paquier and Tom Lane http://git.postgresql.org/pg/commitdiff/732b33f8ae4ecc9d7a9f07fd4cb74a60a2a5e2c2
  • Prevent improper reordering of antijoins vs. outer joins. An outer join appearing within the RHS of an antijoin can't commute with the antijoin, but somehow I missed teaching make_outerjoininfo() about that. In Teodor Sigaev's recent trouble report, this manifests as a "could not find RelOptInfo for given relids" error within eqjoinsel(); but I think silently wrong query results are possible too, if the planner misorders the joins and doesn't happen to trigger any internal consistency checks. It's broken as far back as we had antijoins, so back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/3cf8686014f91174018f20e01dbb0dafdcad0473
  • Add comments warning against generalizing default_with_oids. pg_dump has historically assumed that default_with_oids affects only plain tables and not other relkinds. Conceivably we could make it apply to some newly invented relkind if we did so from the get-go, but changing the behavior for existing object types will break existing dump scripts. Add code comments warning about this interaction. Also, make sure that default_with_oids doesn't cause parse_utilcmd.c to think that CREATE FOREIGN TABLE will create an OID column. I think this is only a latent bug right now, since we don't allow UNIQUE/PKEY constraints in CREATE FOREIGN TABLE, but it's better to be consistent and future-proof. http://git.postgresql.org/pg/commitdiff/0bd11d9711b88e72d2022e25b9227c480aca4978
  • Fix typo in linux startup script. Missed a "$" in what was meant to be a variable substitution. Careless mistake in commit f23425fa950fec3aff458de117037c9caadbc35c. http://git.postgresql.org/pg/commitdiff/f320cbb615e0374b18836337713239da58705cf3

Noah Misch a poussé :

Andrew Dunstan a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Jeff Janes sent in a patch to help make vacuum truncation scans faster.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT IGNORE (and UPDATE).
  • Michael Paquier sent in another revision of a patch to ensure that logical decoding follows timelines.
  • Heikki Linnakangas sent in another revision of a patch to implement access menthods for sequences.
  • Michael Paquier sent in two more revisions of a patch to support TAP tests with MSVC and Windows.
  • SAWADA Masahiko sent in another revision of a patch to help avoid freezing very large tables.
  • Etsuro Fujita sent in another revision of a patch to allow foreign tables to be created with OIDs.
  • Michael Paquier sent in another revision of a patch to ensure that Install.bat works when the target directory contains a space.
  • Michael Paquier sent in another revision of a patch to add pg_settings.pending_restart.
  • Kaigai Kouhei sent in another revision of a patch to add a custom join API.
  • Jan de Visser sent in another revision of a patch to let pg_ctl check the result of a postmaster config reload.
  • Petr Jelinek sent in a patch to add catalog regression tests for the TABLESAMPLE feature.
  • Michael Paquier sent in a patch to fix a comment in src/backend/access/transam/xlog.c.
  • Michael Paquier sent in a patch to allow releasing LWLocks on failure in cases where that makes sense.
  • Amit Kapila sent in another revision of a patch to implement parallel sequential scan.
  • Abhijit Menon-Sen sent in two more revisions of a patch to create a fast bloat measurement tool.
  • Fabrízio de Royes Mello sent in another revision of a patch to add CINE for ALTER TABLE ... ADD COLUMN.
  • David Steele sent in another revision of a patch to add pg_audit.
  • Kyotaro HORIGUCHI sent in two revisions of a patch to fix an issue where coerce_type() fails for Var of type UNKNOWNOID.
  • Andres Freund sent in another revision of a patch to implement replication identifiers.
  • Michael Paquier sent in a patch to fix broken collate.linux.utf8 test coverage.
  • Pavel Stehule sent in two more revisions of a patch to tune the displayability of context messages in PL/pgSQL errors.
  • Fabien COELHO sent in another revision of a patch to extend pgbench expressions with functions.
  • Fabien COELHO sent in another revision of a patch to fix pgbench --progress report under (very) low rate.
  • Euler Taveira de Oliveira sent in a patch to fix some issues in the parallel mode and parallel contexts.

par N Bougain le vendredi 1 mai 2015 à 14h17

samedi 25 avril 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 19 avril 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Peter Eisentraut a poussé :

Fujii Masao a poussé :

  • Silence gettext warning about '\r' escape sequence in translatable string. gettext was unhappy about the commit b216ad7 because it revealed the problem that internationalized messages may contain '\r' escape sequence in pg_rewind. This commit moves '\r' to a separate printf() call. Michael Paquier, bug reported by Peter Eisentraut http://git.postgresql.org/pg/commitdiff/1f94bec7a9e3e6b4fa5468236cf531fec16d1093

Heikki Linnakangas a poussé :

  • Refactor and fix TAP tests of pg_rewind * Don't pass arguments to prove, since that's not supported on perl 5.8 which is the minimum version supported by the TAP tests. Refactor the test files themselves to run the tests twice, in both local and remote mode. * Use eq rather than == for string comparison. This thinko caused the remote versions of the tests to never run. * Add "use strict" and "use warnings", and fix warnings that that produced. * Increase the delay after standby promotion, to make the tests more robust. * In remote mode, the connection string to the promoted standby was incorrect, leading to connection errors. Patch by Michael Paquier, to address Peter Eisentraut's report. http://git.postgresql.org/pg/commitdiff/53ba10770a315361770efdc17d2c01f6a30e3e3d
  • Fix pg_rewind regression tests in VPATH builds. Should call just "pg_rewind", instead of "./pg_rewind". The tests are called so that PATH contains the temporariy installation bin dir. Per report from Alvaro Herrera http://git.postgresql.org/pg/commitdiff/b22a36a62ce312c1df9477382d1da602b0c24f6f
  • Reorganize our CRC source files again. Now that we use CRC-32C in WAL and the control file, the "traditional" and "legacy" CRC-32 variants are not used in any frontend programs anymore. Move the code for those back from src/common to src/backend/utils/hash. Also move the slicing-by-8 implementation (back) to src/port. This is in preparation for next patch that will add another implementation that uses Intel SSE 4.2 instructions to calculate CRC-32C, where available. http://git.postgresql.org/pg/commitdiff/4f700bcd20c087f60346cb8aefd0e269be8e2157
  • Don't archive bogus recycled or preallocated files after timeline switch. After a timeline switch, we would leave behind recycled WAL segments that are in the future, but on the old timeline. After promotion, and after they become old enough to be recycled again, we would notice that they don't have a .ready or .done file, create a .ready file for them, and archive them. That's bogus, because the files contain garbage, recycled from an older timeline (or prealloced as zeros). We shouldn't archive such files. This could happen when we're following a timeline switch during replay, or when we switch to new timeline at end-of-recovery. To fix, whenever we switch to a new timeline, scan the data directory for WAL segments on the old timeline, but with a higher segment number, and remove them. Those don't belong to our timeline history, and are most likely bogus recycled or preallocated files. They could also be valid files that we streamed from the primary ahead of time, but in any case, they're not needed to recover to the new timeline. http://git.postgresql.org/pg/commitdiff/b2a5545bd63fc94a71b1e97ecdd03c605d97a438
  • Use Intel SSE 4.2 CRC instructions where available. Modern x86 and x86-64 processors with SSE 4.2 support have special instructions, crc32b and crc32q, for calculating CRC-32C. They greatly speed up CRC calculation. Whether the instructions can be used or not depends on the compiler and the target architecture. If generation of SSE 4.2 instructions is allowed for the target (-msse4.2 flag on gcc and clang), use them. If they are not allowed by default, but the compiler supports the -msse4.2 flag to enable them, compile just the CRC-32C function with -msse4.2 flag, and check at runtime whether the processor we're running on supports it. If it doesn't, fall back to the slicing-by-8 algorithm. (With the common defaults on current operating systems, the runtime-check variant is what you get in practice.) Abhijit Menon-Sen, heavily modified by me, reviewed by Andres Freund. http://git.postgresql.org/pg/commitdiff/3dc2d62d0486325bf263655c2d9a96aee0b02abe
  • Try to fix the CRC-32C autoconf magic for icc compiler. On gcc and clang, the _mm_crc32_u8 and _mm_crc32_u64 intrinsics are not defined at all, when not building with -msse4.2. But on icc, they are. So we cannot assume that if those intrinsics are defined, we can always use them safely, we might still need the runtime check. To fix, check if the __SSE_4_2__ preprocessor symbol is defined. That's supposed to be defined only when the compiler is targeting a processor that has SSE 4.2 support. Per buildfarm members fulmar and okapi. http://git.postgresql.org/pg/commitdiff/b4eb2d168d2c426978a02de8b9b6ccdb85e1b442
  • Oops, fix misspelled #endif. I hope this fixes the Windows builfarm failures. http://git.postgresql.org/pg/commitdiff/b73e7a0716264e5159947b1a755b9ab864142489
  • Optimize pg_comp_crc32c_sse42 routine slightly, and also use it on x86. Eliminate the separate 'len' variable from the loops, and also use the 4 byte instruction. This shaves off a few more cycles. Even though this routine that uses the special SSE 4.2 instructions is much faster than a generic routine, it's still a hot spot, so let's make it as fast as possible. Change the configure test to not test _mm_crc32_u64. That variant is only available in the 64-bit x86-64 architecture, not in 32-bit x86. Modify pg_comp_crc32c_sse42 so that it only uses _mm_crc32_u64 on x86-64. With these changes, the SSE accelerated CRC-32C implementation can also be used on 32-bit x86 systems. This also fixes the 32-bit MSVC build. http://git.postgresql.org/pg/commitdiff/936546dcbc24ad1f2b3d33e73aa5c5fde4d2be84
  • Fix logic to skip checkpoint if no records have been inserted. After the WAL format changes, the calculation of the size of a checkpoint record became incorrect. Instead of trying to fix the math, check that the previous record, i.e. the xl_prev value that we'd write for the next record, matches the last checkpoint's redo pointer. That way it's not dependent on the size of the checkpoint record at all. The old logic was actually slightly wrong all along: if the previous checkpoint record crossed a page boundary, the page headers threw off the record size calculation, and the checkpoint was not skipped. The new checkpoint would not cross a page boundary, so this only resulted in at most one extra checkpoint after the system became idle. The new logic fixes that. (It's not worth fixing in backbranches). However, it makes some sense to try to keep the latest checkpoint contained fully in a page, or at least in a single WAL segment, just on general robustness grounds. If something goes awfully wrong, it's more likely that you can recover the latest WAL segment, than the last two WAL segments. So I added an extra check that the checkpoint is not skipped if the previous checkpoint crossed a WAL segment. Reported by Jeff Janes. http://git.postgresql.org/pg/commitdiff/3d80a1e0e3e278edc6022d642478dcbd089d4483
  • Shut down test servers after pg_rewind regression tests. Now that the test servers are initialized twice in each .pl script, the single END block is not enough to stop them. Add a new clean_rewind_test function that is called at the end of each test. Michael Paquier http://git.postgresql.org/pg/commitdiff/0d8a22a9ac6a61b7993abb642cb7e4645f4087b0
  • Minor cleanup of pg_rewind. Update comments and function names to use the terms "source" and "target" consistently. Some places were calling them remote and local instead, which was confusing. Fix incorrect comment in extractPageInfo on database creation record - it was wrong on what happens for databases created in the target that don't exist in source. http://git.postgresql.org/pg/commitdiff/41457fcf970f0ec78004cc0f7b29f1d37021fbfb
  • Add missing newlines to error messages. http://git.postgresql.org/pg/commitdiff/b5e384e374657ead815a3393ca59333910611a24
  • Error out in pg_rewind if lstat() fails. A "file not found" is expected if the source server is running, so don't complain about that. But any other error is definitely not expected. http://git.postgresql.org/pg/commitdiff/b5e560c24603e5325a81055c8f36cc45d48609e4
  • Fix assertion failure in logical decoding. Logical decoding set SnapshotData's regd_count field to avoid the snapshot manager from prematurely freeing snapshots that are generated by the decoding system. That was always an abuse of the field, as it was never supposed to be used outside the snapshot manager. Commit 94028691 made snapshot manager's tracking of the snapshots smarter, and that scheme fell apart. The snapshot manager got confused and hit the assertion, when a snapshot that was marked with regd_count==1 was not found in the heap, where the snapshot manager tracks registered the snapshots. To fix, don't abuse the regd_count field like that. Logical decoding still abuses the active_count field for similar purposes, but that's currently harmless. The assertion failure was first reported by Michael Paquier http://git.postgresql.org/pg/commitdiff/e2999abcd14540e66b72deeff75662c1672d7744

Álvaro Herrera a poussé :

Bruce Momjian a poussé :

Stephen Frost a poussé :

  • Fix typo in relcache's equalPolicy(). The USING policies were not being checked for differences as the same policy was being passed in to both sides of the equal(). This could result in backends not realizing that a policy had been changed, if none of the other attributes had been changed. Fix by passing to equal() the policy1 and policy2 using quals for comparison. No need to back-patch as this is not yet released. Noticed while testing changes to RLS proposed by Dean Rasheed. http://git.postgresql.org/pg/commitdiff/ab6d1cd26ebfbfce275cd31af82814c0620e70a2

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Stephen Frost sent in another revision of a patch to use GRANT for access to privileged functions.
  • Etsuro Fujita sent in two more revisions of a patch to fix some oddities with EvalQualPlan for FDW queries involving system columns.
  • David Steele sent in three more revisions of a patch to implement pg_audit.
  • Simon Riggs sent in another revision of a patch to disable HOT cleanup, sometimes.
  • Etsuro Fujita sent in another revision of a patch to fix some infelicities in the new feature of having foreign tables in inheritance trees.
  • Etsuro Fujita sent in a patch to allow WITH OIDs when issuing CREATE FOREIGN TABLE, making it consistently able with ALTER FOREIGN TABLE.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT IGNORE (and UPDATE).
  • Zhang Zq sent in a patch to fix an issue where duplicate checkpoints were being inserted when the system was idle.
  • Qingqing Zhou sent in a patch to assert there is no duplicated exit callbacks.
  • Qingqing Zhou sent in a patch to use outerPlanState() consistently in the executor code.
  • Álvaro Herrera sent in a patch to show xl_prev in xlog.c errcontext.
  • J.L. Tallon sent in a patch to implement a new GUC (default_index_tablespace) plus supporting code.
  • Michael Paquier sent in a doc patch to warn about possible information leakage from compressing full-page writes.
  • Etsuro Fujita sent in a patch to fix some markup in config.sgml.
  • Abhijit Menon-Sen sent in another revision of a patch to recursively fsync PGDATA at startup if needed.
  • Michael Paquier sent in a patch to fix broken Install.bat when target directory contains a space.
  • Shigeru HANADA sent in two more revisions of a patch to add custom and foreign JOIN APIs.
  • Alexander Korotkov sent in another revision of a patch to implement KNN-GiST with recheck.
  • Zhang Zq sent in a patch to implement xidin.
  • Abhijit Menon-Sen sent in a patch to avoid dividing by zero when calculating percentages.
  • Andres Freund sent in another revision of a patch to iIntroduce replication identifiers.
  • Petr Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • Dmitry Dolgov sent in another revision of a patch to add some useful functions for manipulating and pretty-printing JSON.
  • Michael Paquier sent in a patch to add a missing --debug for remote mode in pg_rewind tests to RewindTest.pm.
  • Michael Paquier sent in another revision of a patch to add support for TAP tests on Windows and prefer IPC's run to system_or_bail in pg_rewind tests.
  • Mikko Tiihonen sent in a patch to libpq which allows specifying multiple host names to try to connect to.
  • Tomas Vondra sent in a patch to allow initdb-time support for various compression algorithms.

par N Bougain le samedi 25 avril 2015 à 11h47

dimanche 19 avril 2015

Guillaume Lelarge

Analyse du VACUUM

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

Et voilà, deux nouveaux chapitres écrits, dont un sur le système transactionnel de PostgreSQL. Ce dernier m'a demandé d' étudier plus attentivement le travail de l'opération VACUUM. J'en connaissais le principe et son fonctionnement, à savoir un fonctionnement en trois phases : recherche des éléments à flagguer comme invisibles, suppression de ces éléments dans les index, puis suppression dans la table (pas physiquement). Cependant, je ne l'avais pas regardé plus précisément.

J'ai donc lu le code, puis écrit un petit patch pour mieux suivre cela (disponible en pièce jointe). J'ai exécuté un script SQL pour visualiser différents comportements. Par exemple, on ajoute dix lignes dans une nouvelle table, puis on met à jour une ligne sur trois, et enfin on exécute un VACUUM sur cette table :

CREATE TABLE t2(c1 integer);
ALTER TABLE t2 SET (autovacuum_enabled=off);
INSERT INTO t2 SELECT generate_series(1, 10);
UPDATE t2 SET c1=-c1 where c1%3=1;
SET client_min_messages to log;
VACUUM t2;

Voici le log fourni par le patch :

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82231
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82231) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 1 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 4 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 11
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 12
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 7 is now REDIRECTed to item 13
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 10 is now REDIRECTed to item 14
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (10 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 320)

Il n'y a là qu'une seule étape exécutée. En effet, dû au très petit nombre de lignes dans la table, le seul bloc de 8 Ko n'a pas été entièrement occupé. Du coup, PostgreSQL place les nouvelles versions des lignes mises à jour dans le même bloc que les anciennes versions et utilise le Heap Over Tuple pour lier les enregistrements. De plus, comme il n'y a pas d'index, pas besoin de les mettre à jour.

Maintenant, faisons la même chose avec 400 lignes (en fait, suffisamment pour remplir plus d'un bloc). Les logs sont beaucoup plus importants.

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82234
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82234) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 3 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 76 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 1 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 4 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 223 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 226 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (150 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 4800)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 1 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 4 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 223 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 226 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 1 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 4 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 223 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 226 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (150 have storage, 76 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 4800)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 58 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 3 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 6 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 171 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 174 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (168 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5376)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 3 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 6 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 9 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 171 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 174 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 3 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 6 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 171 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 174 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (168 have storage, 58 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5376)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 2
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 0 deleted items found

Chaque élément devenu invisible est déclaré DEAD lors de la première étape, puis UNUSED à la troisième étape.

Il est dit que le fillfactor permet d'augmenter l'utilisation du Heap Over Tuple. Voici ce que cela donne avec un fillfactor à 90% :

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82237
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82237) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 3 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 68 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 205
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 206
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 61 is now REDIRECTed to item 225
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 64 is now REDIRECTed to item 226
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 67 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 70 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 199 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 202 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (158 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5056)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 67 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 70 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 199 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 202 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 67 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 70 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 199 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 202 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (158 have storage, 46 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5056)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 66 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 205
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 206
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 61 is now REDIRECTed to item 225
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 64 is now REDIRECTed to item 226
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 67 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 70 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 193 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 196 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (160 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5120)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 67 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 70 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 193 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 196 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 67 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 70 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 193 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 196 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (160 have storage, 44 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5120)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 2
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 0 deleted items found

Certains enregistrements bénéficient de HOT, mais la majorité deviennent UNUSED. Essayons avec un fillfactor de 50% :

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82240
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82240) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 4 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 38 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 114
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 115
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 109 is now REDIRECTed to item 150
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 112 is now REDIRECTed to item 151
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (113 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 3616)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 38 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 2 is now REDIRECTed to item 114
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 5 is now REDIRECTed to item 115
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 110 is now REDIRECTed to item 150
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 113 is now REDIRECTed to item 151
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (113 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 3616)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 2
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 37 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 3 is now REDIRECTed to item 114
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 6 is now REDIRECTed to item 115
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 108 is now REDIRECTed to item 149
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 111 is now REDIRECTed to item 150
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (113 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 3616)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 3
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 21 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 62
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 63
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 58 is now REDIRECTed to item 81
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 61 is now REDIRECTed to item 82
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (61 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 1952)

Dans ce cas, tous les enregistrements bénéficient de HOT, aucun n'est UNUSED. Cela permet des mises à jour et une maintenance plus rapides, mais c'est au prix d'une table plus volumineuse (sur disque et dans le cache des relations de PostgreSQL).

Et du coup, vous vous demandez peut-être quand va sortir la version beta 0.3 du livre ? D'ici peu a priori. Un peu de relecture, quelques ajustements de dernières minutes, et ça devrait être prêt :)

par Guillaume Lelarge le dimanche 19 avril 2015 à 17h21

mardi 14 avril 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 12 avril 2015

L'appel à conférenciers pour le PGDay de Belfort (France) se termine le 13 avril 2015. La conférence aura lieu le 2 juin : http://select-2-6-2015-as-pgday.org

La 4ème conférence PostgreSQL turque se tiendra à Istanbul le 9 mai 2015 : http://pgday.postgresql.org.tr/en/index.html

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Fujii Masao a poussé :

Álvaro Herrera a poussé :

  • Fix object identities for pg_conversion objects. This was already fixed in 0d906798f, but I failed to update the array-formatted case. This is not backpatched, since this only affects the code path introduced by commit a676201490c. http://git.postgresql.org/pg/commitdiff/70dc2db7f1dfdecdacf595bf00964cb20ad5a835
  • pg_event_trigger_dropped_objects: add is_temp column. It now also reports temporary objects dropped that are local to the backend. Previously we weren't reporting any temp objects because it was deemed unnecessary; but as it turns out, it is necessary if we want to keep close track of DDL command execution inside one session. Temp objects are reported as living in schema pg_temp, which works because such a schema-qualification always refers to the temp objects of the current session. http://git.postgresql.org/pg/commitdiff/e9a077cad3799b41e8deef6fd8cb87e50164a791
  • Remove variable shadowing. Commit a2e35b53 should have removed the variable declaration in the inner block, but didn't. As a result, the returned address might end up not being what was intended. http://git.postgresql.org/pg/commitdiff/4e17e32f53c2de4a862ee5ef8bdcfa9152c11e25
  • Fix autovacuum launcher shutdown sequence. It was previously possible to have the launcher re-execute its main loop before shutting down if some other signal was received or an error occurred after getting SIGTERM, as reported by Qingqing Zhou. While investigating, Tom Lane further noticed that if autovacuum had been disabled in the config file, it would misbehave by trying to start a new worker instead of bailing out immediately -- it would consider itself as invoked in emergency mode. Fix both problems by checking the shutdown flag in a few more places. These problems have existed since autovacuum was introduced, so backpatch all the way back. http://git.postgresql.org/pg/commitdiff/5df64f298d2863c9fb39437abb3ae6f988aedc0a
  • Change SQLSTATE for event triggers "wrong context" message. When certain event-trigger-only functions are called when not in the wrong context, they were reporting the "feature not supported" SQLSTATE, which is somewhat misleading. Create a new custom error code for such uses instead. Not backpatched since it may be seen as an undesirable behavioral change. Author: Michael Paquier. Discussion: https://www.postgresql.org/message-id/CAB7nPqQ-5NAkHQHh_NOm7FPep37NCiLKwPoJ2Yxb8TDoGgbYYA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/73206812cd97436cffd8f331dbb09d38a2728162
  • Optimize locking a tuple already locked by another subxact. Locking and updating the same tuple repeatedly led to some strange multixacts being created which had several subtransactions of the same parent transaction holding locks of the same strength. However, once a subxact of the current transaction holds a lock of a given strength, it's not necessary to acquire the same lock again. This made some coding patterns much slower than required. The fix is twofold. First we change HeapTupleSatisfiesUpdate to return HeapTupleBeingUpdated for the case where the current transaction is already a single-xid locker for the given tuple; it used to return HeapTupleMayBeUpdated for that case. The new logic is simpler, and the change to pgrowlocks is a testament to that: previously we needed to check for the single-xid locker separately in a very ugly way. That test is simpler now. As fallout from the HTSU change, some of its callers need to be amended so that tuple-locked-by-own-transaction is taken into account in the BeingUpdated case rather than the MayBeUpdated case. For many of them there is no difference; but heap_delete() and heap_update now check explicitely and do not grab tuple lock in that case. The HTSU change also means that routine MultiXactHasRunningRemoteMembers introduced in commit 11ac4c73cb895 is no longer necessary and can be removed; the case that used to require it is now handled naturally as result of the changes to heap_delete and heap_update. The second part of the fix to the performance issue is to adjust heap_lock_tuple to avoid the slowness: 1. Previously we checked for the case that our own transaction already held a strong enough lock and returned MayBeUpdated, but only in the multixact case. Now we do it for the plain Xid case as well, which saves having to LockTuple. 2. If the current transaction is the only locker of the tuple (but with a lock not as strong as what we need; otherwise it would have been caught in the check mentioned above), we can skip sleeping on the multixact, and instead go straight to create an updated multixact with the additional lock strength. 3. Most importantly, make sure that both the single-xid-locker case and the multixact-locker case optimization are applied always. We do this by checking both in a single place, rather than them appearing in two separate portions of the routine -- something that is made possible by the HeapTupleSatisfiesUpdate API change. Previously we would only check for the single-xid case when HTSU returned MayBeUpdated, and only checked for the multixact case when HTSU returned BeingUpdated. This was at odds with what HTSU actually returned in one case: if our own transaction was locker in a multixact, it returned MayBeUpdated, so the optimization never applied. This is what led to the large multixacts in the first place. Per bug report #8470 by Oskari Saarenmaa. http://git.postgresql.org/pg/commitdiff/27846f02c176eebe7e08ce51ed4d52140454e196

Simon Riggs a poussé :

Heikki Linnakangas a poussé :

Tom Lane a poussé :

Robert Haas a poussé :

Bruce Momjian a poussé :

Andres Freund a poussé :

Magnus Hagander a poussé :

Peter Eisentraut a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Fabrízio de Royes Mello sent in two more revisions of a patch to refactor reloptions to set locklevel.
  • Artem Luzyanin sent in two revisions of a patch to consolidate the documentation of spinlocks and like kind items.
  • Emre Hasegeli sent in another revision of a patch to add a BRIN range operator class.
  • Tomas Vondra sent in a patch to use foreign keys to improve join estimates.
  • Michael Paquier sent in a patch to ignore some binaries generated in src/test.
  • SAWADA Masahiko and Fabrízio de Royes Mello traded patches to add REINDEX ... VERBOSE.
  • Shigeru HANADA sent in four more revisions of a patch to add a foreign join API.
  • Petr Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • Craig Ringer sent in a patch to add a pid column to pg_replication_slots.
  • Peter Eisentraut and Pavel Stehule traded patches to add TRANSFORMS.
  • Michael Paquier sent in another revision of a patch to add an error code to track unsupported contexts.
  • Álvaro Herrera sent in two more revisions of a patch to add deparsing utilities.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT UPDATE (and IGNORE).
  • Tom Lane sent in another revision of a patch to implement UPDATE (*) SET ...
  • Fujii Masao sent in another revision of a patch to remove the obsolete FORCE option from REINDEX.
  • Dean Rasheed sent in another revision of a patch to fix some infelicities in the error reporting for row-level access control.
  • Kyotaro HORIGUCHI sent in another revision of a patch to implement regnamespace and regrole.
  • Antonin Houska sent in a patch to fix some issues in xlogreader.
  • Craig Ringer sent in two revisions of a patch to make pg_dump -t take materialized views, matview data, foreign tables, and sequences.
  • Bruce Momjian sent in two revisions of a patch to ensure that CREATE TABLE (LIKE...) preserves the relhasoids setting.
  • Michael Paquier sent in another revision of a patch to suppport TAP tests with MSVC and Windows.
  • Jan Urbański sent in another revision of a patch to fix a bug in libpq's multi-threaded SSL callback handling.
  • Ian Stakenvicius sent in a patch to fix an issue on Gentoo where postgres fails to start with timezone-data >=2013e.
  • Etsuro Fujita sent in another revision of a patch to fix a problem where EvalPlanQual behaves oddly for FDW queries involving system columns.
  • Pavel Stehule sent in a patch to add a "raw" output option to COPY.
  • Chen Huajun sent in a patch to prevent setting Win32 server-side socket buffer size on Windows 2012.
  • David Rowley sent in a patch to fix a few appendStringInfo* calls that were not quite doing things the way as intended.
  • Michael Paquier sent in another revision of a patch to help improve the performance of make check-world.
  • Heikki Linnakangas sent in a patch to remove xlogrecord padding.

par N Bougain le mardi 14 avril 2015 à 21h05

samedi 11 avril 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 5 avril 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Fix multiple bugs and infelicities in pg_rewind. Bugs all spotted by Coverity, including wrong realloc() size request and memory leaks. Cosmetic improvements by me. The usage of the global variable "filemap" here is still pretty awful, but at least I got rid of the gratuitous aliasing in several routines (which was helping to annoy Coverity, as well as being a bug risk). http://git.postgresql.org/pg/commitdiff/c67f366fa9f748257861ee233b47b80eb5ffa857
  • Clean up all the cruft after a pg_rewind test run. regress_log temp directory was properly .gitignore'd, which may explain why it got left out of the "make clean" action. http://git.postgresql.org/pg/commitdiff/1c41e2a998a0de16d9d33949a7b98a5be3d2477c
  • Fix rare core dump in BackendIdGetTransactionIds(). BackendIdGetTransactionIds() neglected the possibility that the PROC pointer in a ProcState array entry is null. In current usage, this could only crash if the other backend had exited since pgstat_read_current_status saw it as active, which is a pretty narrow window. But it's reachable in the field, per bug #12918 from Vladimir Borodin. Back-patch to 9.4 where the faulty code was introduced. http://git.postgresql.org/pg/commitdiff/701dcc983eb4d08dd36bb3a0ddba255819797760
  • Be more careful about printing constants in ruleutils.c. The previous coding in get_const_expr() tried to avoid quoting integer, float, and numeric literals if at all possible. While that looks nice, it means that dumped expressions might re-parse to something that's semantically equivalent but not the exact same parsetree; for example a FLOAT8 constant would re-parse as a NUMERIC constant with a cast to FLOAT8. Though the result would be the same after constant-folding, this is problematic in certain contexts. In particular, Jeff Davis pointed out that this could cause unexpected failures in ALTER INHERIT operations because of child tables having not-exactly-equivalent CHECK expressions. Therefore, favor correctness over legibility and dump such constants in quotes except in the limited cases where they'll be interpreted as the same type even without any casting. This results in assorted small changes in the regression test outputs, and will affect display of user-defined views and rules similarly. The odds of that causing problems in the field seem non-negligible; given the lack of previous complaints, it seems best not to change this in the back branches. http://git.postgresql.org/pg/commitdiff/542320c2bd0b3796a8a9a4617cdb23fbad473390
  • Fix bogus concurrent use of _hash_getnewbuf() in bucket split code. _hash_splitbucket() obtained the base page of the new bucket by calling _hash_getnewbuf(), but it held no exclusive lock that would prevent some other process from calling _hash_getnewbuf() at the same time. This is contrary to _hash_getnewbuf()'s API spec and could in fact cause failures. In practice, we must only call that function while holding write lock on the hash index's metapage. An additional problem was that we'd already modified the metapage's bucket mapping data, meaning that failure to extend the index would leave us with a corrupt index. Fix both issues by moving the _hash_getnewbuf() call to just before we modify the metapage in _hash_expandtable(). Unfortunately there's still a large problem here, which is that we could also incur ENOSPC while trying to get an overflow page for the new bucket. That would leave the index corrupt in a more subtle way, namely that some index tuples that should be in the new bucket might still be in the old one. Fixing that seems substantially more difficult; even preallocating as many pages as we could possibly need wouldn't entirely guarantee that the bucket split would complete successfully. So for today let's just deal with the base case. Per report from Antonin Houska. Back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/ed9cc2b5df59fdbc50cce37399e26b03ab2c1686
  • Fix incorrect markup in documentation of window frame clauses. You're required to write either RANGE or ROWS to start a frame clause, but the documentation incorrectly implied this is optional. Noted by David Johnston. http://git.postgresql.org/pg/commitdiff/f6caf5acf1def92d7425151a92fd990c566fdcc3
  • Provide real selectivity estimators for inet/cidr operators. This patch fills in the formerly-stub networksel() and networkjoinsel() estimation functions. Those are used for << <<= >> >>= and && operators on inet/cidr types. The estimation is not perfect, certainly, because we rely on the existing statistics collected for the inet btree operators. But it's a long way better than nothing, and it's not clear that asking ANALYZE to collect separate stats for these operators would be a win. Emre Hasegeli, with reviews from Dilip Kumar and Heikki Linnakangas, and some further hacking by me http://git.postgresql.org/pg/commitdiff/89840d7d3fa943cb932f6a00707fdb17a9cab001
  • Fix rare startup failure induced by MVCC-catalog-scans patch. While a new backend nominally participates in sinval signaling starting from the SharedInvalBackendInit call near the top of InitPostgres, it cannot recognize sinval messages for unshared catalogs of its database until it has set up MyDatabaseId. This is not problematic for the catcache or relcache, which by definition won't have loaded any data from or about such catalogs before that point. However, commit 568d4138c646cd7c introduced a mechanism for re-using MVCC snapshots for catalog scans, and made invalidation of those depend on recognizing relevant sinval messages. So it's possible to establish a catalog snapshot to read pg_authid and pg_database, then before we set MyDatabaseId, receive sinval messages that should result in invalidating that snapshot --- but do not, because we don't realize they are for our database. This mechanism explains the intermittent buildfarm failures we've seen since commit 31eae6028eca4365. That commit was not itself at fault, but it introduced a new regression test that does reconnections concurrently with the "vacuum full pg_am" command in vacuum.sql. This allowed the pre-existing error to be exposed, given just the right timing, because we'd fail to update our information about how to access pg_am. In principle any VACUUM FULL on a system catalog could have created a similar hazard for concurrent incoming connections. Perhaps there are more subtle failure cases as well. To fix, force invalidation of the catalog snapshot as soon as we've set MyDatabaseId. Back-patch to 9.4 where the error was introduced. http://git.postgresql.org/pg/commitdiff/bc49d9324a464fce8f60e1bc14531631883021d4
  • Remove unnecessary variables in _hash_splitbucket(). Commit ed9cc2b5df59fdbc50cce37399e26b03ab2c1686 made it unnecessary to pass start_nblkno to _hash_splitbucket(), and for that matter unnecessary to have the internal nblkno variable either. My compiler didn't complain about that, but some did. I also rearranged the use of oblkno a bit to make that case more parallel. Report and initial patch by Petr Jelinek, rearranged a bit by me. Back-patch to all branches, like the previous patch. http://git.postgresql.org/pg/commitdiff/b7e1652d5de8b618c0204588969c8b59d12e9361
  • Fix TAP tests to use only standard command-line argument ordering. Some of the TAP tests were supposing that PG programs would accept switches after non-switch arguments on their command lines. While GNU getopt_long() does allow that, our own implementation does not, and it's nowhere suggested in our documentation that such cases should work. Adjust the tests to use only the documented syntax. Back-patch to 9.4, since without this the TAP tests fail when run with src/port's getopt_long() implementation. Michael Paquier http://git.postgresql.org/pg/commitdiff/c67a86f7da90c30b81f91957023fb752f06f0598
  • Fix incorrect matching of subexpressions in outer-join plan nodes. Previously we would re-use input subexpressions in all expression trees attached to a Join plan node. However, if it's an outer join and the subexpression appears in the nullable-side input, this is potentially incorrect for apparently-matching subexpressions that came from above the outer join (ie, targetlist and qpqual expressions), because the executor will treat the subexpression value as NULL when maybe it should not be. The case is fairly hard to hit because (a) you need a non-strict subexpression (else NULL is correct), and (b) we don't usually compute expressions in the outputs of non-toplevel plan nodes. But we might do so if the expressions are sort keys for a mergejoin, for example. Probably in the long run we should make a more explicit distinction between Vars appearing above and below an outer join, but that will be a major planner redesign and not at all back-patchable. For the moment, just hack set_join_references so that it will not match any non-Var expressions coming from nullable inputs to expressions that came from above the join. (This is somewhat overkill, in that a strict expression could still be matched, but it doesn't seem worth the effort to check that.) Per report from Qingqing Zhou. The added regression test case is based on his example. This has been broken for a very long time, so back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/ca6805338fba010cc3f8b842905d7a62e280b7ab
  • Suppress clang's unhelpful gripes about -pthread switch being unused. Considering the number of cases in which "unused" command line arguments are silently ignored by compilers, it's fairly astonishing that anybody thought this warning was useful; it's certainly nothing but an annoyance when building Postgres. One such case is that neither gcc nor clang complain about unrecognized -Wno-foo switches, making it more difficult to figure out whether the switch does anything than one could wish. Back-patch to 9.3, which is as far back as the patch applies conveniently (we'd have to back-patch PGAC_PROG_CC_VAR_OPT to go further, and it doesn't seem worth that). http://git.postgresql.org/pg/commitdiff/73b416b2e41237b657d29d8f42a4bb34bf700928

Heikki Linnakangas a poussé :

Álvaro Herrera a poussé :

  • Fix lost persistence setting during REINDEX INDEX. ReindexIndex() trusts a parser-built RangeVar with the persistence to use for the new copy of the index; but the parser naturally does not know what's the persistence of the original index. To find out the correct persistence, grab it from relcache. This bug was introduced by commit 85b506bbfc2937c9, and therefore no backpatch is necessary. Bug reported by Thom Brown, analysis and patch by Michael Paquier; test case provided by Fabrízio de Royes Mello. http://git.postgresql.org/pg/commitdiff/0853630159944bb3652336602ff5f7f62cd27a5a
  • Change array_offset to return subscripts, not offsets ... and rename it and its sibling array_offsets to array_position and array_positions, to account for the changed behavior. Having the functions return subscripts better matches existing practice, and is better suited to using the result value as a subscript into the array directly. For one-based arrays, the new definition is identical to what was originally committed. (We use the term "subscript" in the documentation, which is what we use whenever we talk about arrays; but the functions themselves are named using the word "position" to match the standard-defined POSITION() functions.) Author: Pavel Stěhule. Behavioral problem noted by Dean Rasheed. http://git.postgresql.org/pg/commitdiff/97690ea6e86c412461dd5dc99953b829564d1a55
  • psql: fix \connect with URIs and conninfo strings. psql was already accepting conninfo strings as the first parameter in \connect, but the way it worked wasn't sane; some of the other parameters would get the previous connection's values, causing it to connect to a completely unexpected server or, more likely, not finding any server at all because of completely wrong combinations of parameters. Fix by explicitely checking for a conninfo-looking parameter in the dbname position; if one is found, use its complete specification rather than mix with the other arguments. Also, change tab-completion to not try to complete conninfo/URI-looking "dbnames" and document that conninfos are accepted as first argument. There was a weak consensus to backpatch this, because while the behavior of using the dbname as a conninfo is nowhere documented for \connect, it is reasonable to expect that it works because it does work in many other contexts. Therefore this is backpatched all the way back to 9.0. To implement this, routines previously private to libpq have been duplicated so that psql can decide what looks like a conninfo/URI string. In back branches, just duplicate the same code all the way back to 9.2, where URIs where introduced; 9.0 and 9.1 have a simpler version. In master, the routines are moved to src/common and renamed. Author: David Fetter, Andrew Dunstan. Some editorialization by me (probably earning a Gierth's "Sloppy" badge in the process.) Reviewers: Andrew Gierth, Erik Rijkers, Pavel Stěhule, Stephen Frost, Robert Haas, Andrew Dunstan. http://git.postgresql.org/pg/commitdiff/fcef1617295c074f2684c887627184d2fc26ac04
  • psql: fix \connect with URIs and conninfo strings. This is the second try at this, after fcef1617295 failed miserably and had to be reverted: as it turns out, libpq cannot depend on libpgcommon after all. Instead of shuffling code in the master branch, make that one just like 9.4 and accept the duplication. (This was all my own mistake, not the patch submitter's). psql was already accepting conninfo strings as the first parameter in \connect, but the way it worked wasn't sane; some of the other parameters would get the previous connection's values, causing it to connect to a completely unexpected server or, more likely, not finding any server at all because of completely wrong combinations of parameters. Fix by explicitely checking for a conninfo-looking parameter in the dbname position; if one is found, use its complete specification rather than mix with the other arguments. Also, change tab-completion to not try to complete conninfo/URI-looking "dbnames" and document that conninfos are accepted as first argument. There was a weak consensus to backpatch this, because while the behavior of using the dbname as a conninfo is nowhere documented for \connect, it is reasonable to expect that it works because it does work in many other contexts. Therefore this is backpatched all the way back to 9.0. Author: David Fetter, Andrew Dunstan. Some editorialization by me (probably earning a Gierth's "Sloppy" badge in the process.) Reviewers: Andrew Gierth, Erik Rijkers, Pavel Stěhule, Stephen Frost, Robert Haas, Andrew Dunstan. http://git.postgresql.org/pg/commitdiff/e146ca682062ca1f5015f3820571c5359f5f9dba
  • autovacuum: Fix polarity of "wraparound" variable. Commit 0d831389749a3 inadvertently reversed the meaning of the wraparound variable. This causes vacuums which are not required for wraparound to wait for locks to be acquired, and what is worse, it allows wraparound vacuums to skip locked pages. Bug reported by Jeff Janes in http://www.postgresql.org/message-id/CAMkU=1xmTEiaY=5oMHsSQo5vd9V1Ze4kNLL0qN2eH0P_GXOaYw@mail.gmail.com Analysis and patch by Kyotaro HORIGUCHI http://git.postgresql.org/pg/commitdiff/00ee6c7672fe0bf9448bc744b5e3408f5ebffc2e
  • Have autovacuum workers listen to SIGHUP, too. They have historically ignored it, but it's been said to be useful at times to change their settings mid-flight. Author: Michael Paquier http://git.postgresql.org/pg/commitdiff/a75fb9b335db0e063ece283ebd207530abe1b53b
  • Add log_min_autovacuum_duration per-table option. This is useful to control autovacuum log volume, for situations where monitoring only a set of tables is necessary. Author: Michael Paquier Reviewed by: A team led by Naoya Anzai (also including Akira Kurosawa, Taiki Kondo, Huong Dangminh), Fujii Masao. http://git.postgresql.org/pg/commitdiff/4ff695b17d32a9c330952192dbc789d31a5e2f5e
  • Transform ALTER TABLE/SET TYPE/USING expr during parse analysis. This lets later stages have access to the transformed expression; in particular it allows DDL-deparsing code during event triggers to pass the transformed expression to ruleutils.c, so that the complete command can be deparsed. This shuffles the timing of the transform calls a bit: previously, nothing was transformed during parse analysis, and only the RELKIND_RELATION case was being handled during execution. After this patch, all expressions are transformed during parse analysis (including those for relkinds other than RELATION), and the error for other relation kinds is thrown only during execution. So we do more work than before to reject some bogus cases. That seems acceptable. http://git.postgresql.org/pg/commitdiff/9550e8348b7965715789089555bb5a3fda8c269c

Andrew Dunstan a poussé :

  • Run pg_upgrade and pg_resetxlog with restricted token on Windows. As with initdb these programs need to run with a restricted token, and if they don't pg_upgrade will fail when run as a user with Adminstrator privileges. Backpatch to all live branches. On the development branch the code is reorganized so that the restricted token code is now in a single location. On the stable bramches a less invasive change is made by simply copying the relevant code to pg_upgrade.c and pg_resetxlog.c. Patches and bug report from Muhammad Asif Naeem, reviewed by Michael Paquier, slightly edited by me. http://git.postgresql.org/pg/commitdiff/fa1e5afa8a26d467aec7c8b36a0b749b690f636c
  • Enable float8-byval as the default for 64 bit MSVC builds. This is a long-standing inconsistency that was probably just missed when we got 64 bit MSVC builds. This brings the platform into line with all other systems. http://git.postgresql.org/pg/commitdiff/cf376a4adc0805b0960a5f8e8325fae7d4456926

Bruce Momjian a poussé :

Fujii Masao a poussé :

Simon Riggs a poussé :

Robert Haas a poussé :

Andres Freund a poussé :

  • Define integer limits independently from the system definitions. In 83ff1618 we defined integer limits iff they're not provided by the system. That turns out not to be the greatest idea because there's different ways some datatypes can be represented. E.g. on OSX PG's 64bit datatype will be a 'long int', but OSX unconditionally uses 'long long'. That disparity then can lead to warnings, e.g. around printf formats. One way to fix that would be to back int64 using stdint.h's int64_t. While a good idea it's not that easy to implement. We would e.g. need to include stdint.h in our external headers, which we don't today. Also computing the correct int64 printf formats in that case is nontrivial. Instead simply prefix the integer limits with PG_ and define them unconditionally. I've adjusted all the references to them in code, but not the ones in comments; the latter seems unnecessary to me. Discussion: 20150331141423.GK4878@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/62e2a8dc2c7f6b1351a0385491933af969ed4265

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in a doc patch to describe more precisely the rounding behavior of numeric and double precision.
  • Tomas Vondra sent in another revision of a patch to implement multivariate statistics, useful for among other things cross-column correlations.
  • Heikki Linnakangas sent in a patch to implement SCRAM authentication.
  • David Fetter sent in two more revisions of a patch to allow make_date() to use negative years for BCE dates.
  • Craig Ringer sent in a patch to add views, materialized views, and foreign tables to the meaning of pg_dump's -t command line argument.
  • Fabrízio de Royes Mello sent in a WIP patch to reduce lock level when setting autovacuum reloptions in "ALTER TABLE .. SET ( .. )" statement.
  • Pavel Stehule sent in another revision of a patch to add a row_to_array() function.
  • Michael Paquier sent in two revisions of a patch to add some tests for pg_rewind.
  • Abhijit Menon-Sen sent in two more revisions of a patch to add pgstatbloat to pgstattuple.
  • Haribabu Kommi sent in another revision of a patch to add a catalog view to for pg_hba.conf.
  • Tomas Vondra sent in another revision of a patch to improve the n-distinct estimator.
  • Jehan-Guillaume de Rorthais sent in a patch to document the maximum number of files in the pg_xlog directory.
  • Stephen Frost sent in another revision of a patch to implement role attributes.
  • Bruce Momjian sent in two more revisions of a patch to fix some infelicities in zero-padding in to_char().
  • David Steele sent in another revision of a patch to implement pg_audit.
  • Bruce Momjian sent in two revisions of a patch to fix an infelicity in the SSPI error reporting code.
  • Heikki Linnakangas sent in another revision of a patch to use Intel SSE4.2 CRC instructions where available.
  • Petr Jelinek sent in a patch to remove an unused variable in src/backend/access/hash/hashpage.c.
  • Petr Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • SAWADA Masahiko sent in a WIP patch to allow making read-only tables.
  • Shigeru HANADA sent in another revision of a patch to add join push-down support to postgres_fdw.
  • Etsuro Fujita sent in another revision of a patch to fix some infelicities in the way EvalPlanQual behaves for FDW queries involving system columns.
  • SAWADA Masahiko sent in another revision of a patch to create a pg_file_settings view.
  • Tomas Vondra sent in a patch to fix to get rid of the spurious warnings caused by the recent change to _hash_splitbucket().
  • Peter Geoghegan sent in a patch to make trace_sort control abbreviation debug output for the text opclass, making it consistent with the numeric opclass.

par N Bougain le samedi 11 avril 2015 à 16h46

Nouvelles hebdomadaires de PostgreSQL - 29 mars 2015

Le pgDay Paris 2015 aura lieu le 21 avril à Paris au siège de La Poste. Les inscriptions sont ouvertes : http://pgday.paris/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mars

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Robert Haas a poussé :

Andres Freund a poussé :

  • Fix copy & paste error in 4f1b890b137. Due to the bug delayed standbys would not delay when applying prepared transactions. Discussion: CAB7nPqT6BO1cCn+sAyDByBxA4EKZNAiPi2mFJ=ANeZmnmewRyg@mail.gmail.com Michael Paquier via Coverity. http://git.postgresql.org/pg/commitdiff/a1105c3dd44c1fb76eb62a708f0421f21b9dde9b
  • Don't delay replication for less than recovery_min_apply_delay's resolution. Recovery delays are implemented by waiting on a latch, and latches take milliseconds as a parameter. The required amount of waiting was computed using microsecond resolution though and the wait loop's abort condition was checking the delay in microseconds as well. This could lead to short spurts of busy looping when the overall wait time was below a millisecond, but above 0 microseconds. Instead just formulate the wait loop's abort condition in millisecond granularity as well. Given that that's recovery_min_apply_delay resolution, it seems harmless to not wait for less than a millisecond. Backpatch to 9.4 where recovery_min_apply_delay was introduced. Discussion: 20150323141819.GH26995@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/87cec51d3ad1107f6f224ed7d773e70c8896e4c0
  • Centralize definition of integer limits. Several submitted and even committed patches have run into the problem that C89, our baseline, does not provide minimum/maximum values for various integer datatypes. C99's stdint.h does, but we can't rely on it. Several parts of the code defined limits locally, so instead centralize the definitions to c.h. This patch also changes the more obvious usages of literal limit values; there's more places that could be changed, but it's less clear whether it's beneficial to change those. Author: Andrew Gierth Discussion: 87619tc5wc.fsf@news-spur.riddles.org.uk http://git.postgresql.org/pg/commitdiff/83ff1618bc9d4e530d3ef2a668a71326784a753c

Heikki Linnakangas a poussé :

Álvaro Herrera a poussé :

  • vacuumdb: Check result status of PQsendQuery. Noticed by Coverity http://git.postgresql.org/pg/commitdiff/871293fb7fa869fbe0f202bef2e0d781bfd21b3e
  • Fix gram.y comment to match reality. There are other comments in there that don't precisely match what's implemented, but this one confused me enough to be worth fixing. http://git.postgresql.org/pg/commitdiff/dc8e05295ab126bc4c943cab3e8e117489ecb246
  • Fix bug for array-formatted identities of user mappings. I failed to realize that server names reported in the object args array would get quoted, which is wrong; remove that, making sure that it's only quoted in the string-formatted identity. This bug was introduced by my commit cf34e373, which was backpatched, but since object name/args arrays are new in commit a676201490c8, there is no need to backpatch this any further. http://git.postgresql.org/pg/commitdiff/b3196e65f5bfc997ec7fa3f91645a09289c10dee
  • Add OID output argument to DefineTSConfiguration ... which is set to the OID of a copied text search config, whenever the COPY clause is used. This is in the spirit of commit a2e35b53c39. http://git.postgresql.org/pg/commitdiff/8217fb1441ce4b4e1785f9acfa0ce50039247a10
  • Return ObjectAddress in many ALTER TABLE sub-routines. Since commit a2e35b53c39b2a, most CREATE and ALTER commands return the ObjectAddress of the affected object. This is useful for event triggers to try to figure out exactly what happened. This patch extends this idea a bit further to cover ALTER TABLE as well: an auxiliary ObjectAddress is returned for each of several subcommands of ALTER TABLE. This makes it possible to decode with precision what happened during execution of any ALTER TABLE command; for instance, which constraint was added by ALTER TABLE ADD CONSTRAINT, or which parent got dropped from the parents list by ALTER TABLE NO INHERIT. As with the previous patch, there is no immediate user-visible change here. This is all really just continuing what c504513f83a9ee8 started. Reviewed by Stephen Frost. http://git.postgresql.org/pg/commitdiff/bdc3d7fa2376a7a1e977383cc3221cfe44c4a893

Tom Lane a poussé :

  • Apply table and domain CHECK constraints in name order. Previously, CHECK constraints of the same scope were checked in whatever order they happened to be read from pg_constraint. (Usually, but not reliably, this would be creation order for domain constraints and reverse creation order for table constraints, because of differing implementation details.) Nondeterministic results of this sort are problematic at least for testing purposes, and in discussion it was agreed to be a violation of the principle of least astonishment. Therefore, borrow the principle already established for triggers, and apply such checks in name order (using strcmp() sort rules). This lets users control the check order if they have a mind to. Domain CHECK constraints still follow the rule of checking lower nested domains' constraints first; the name sort only applies to multiple constraints attached to the same domain. In passing, I failed to resist the temptation to wordsmith a bit in create_domain.sgml. Apply to HEAD only, since this could result in a behavioral change in existing applications, and the potential regression test failures have not actually been observed in our buildfarm. http://git.postgresql.org/pg/commitdiff/e5f455f59fed0632371cddacddd79895b148dc07
  • Fix ExecOpenScanRelation to take a lock on a ROW_MARK_COPY relation. ExecOpenScanRelation assumed that any relation listed in the ExecRowMark list has been locked by InitPlan; but this is not true if the rel's markType is ROW_MARK_COPY, which is possible if it's a foreign table. In most (possibly all) cases, failure to acquire a lock here isn't really problematic because the parser, planner, or plancache would have taken the appropriate lock already. In principle though it might leave us vulnerable to working with a relation that we hold no lock on, and in any case if the executor isn't depending on previously-taken locks otherwise then it should not do so for ROW_MARK_COPY relations. Noted by Etsuro Fujita. Back-patch to all active versions, since the inconsistency has been there a long time. (It's almost certainly irrelevant in 9.0, since that predates foreign tables, but the code's still wrong on its own terms.) http://git.postgresql.org/pg/commitdiff/feeb526cfe33657b8aa8b0cdd45b2ef0d9898877
  • Upgrade src/port/rint.c to be POSIX-compliant. The POSIX spec says that rint() rounds halfway cases to nearest even. Our substitute implementation failed to do that, rather rounding halfway cases away from zero; and it also got some other cases (such as minus zero) wrong. This led to observable cross-platform differences, as reported in bug #12885 from Rich Schaaf; in particular, casting from float to int didn't honor round-to-nearest-even on builds using rint.c. Implement something that attempts to cover all cases per spec, and add some simple regression tests so that we'll notice if any platforms still get this wrong. Although this is a bug fix, no back-patch, as a behavioral change in the back branches was agreed not to be a good idea. Pedro Gimeno Fortea, reviewed by Michael Paquier and myself http://git.postgresql.org/pg/commitdiff/06bf0dd6e354c765403d1331cc9896b360754521
  • Add an ASSERT statement in plpgsql. This is meant to make it easier to insert simple debugging cross-checks in plpgsql functions. Pavel Stehule, reviewed by Jim Nasby http://git.postgresql.org/pg/commitdiff/a4847fc3ef139ba9a8ffebb6ffa06ee72078ffa2
  • Suppress some unused-variable complaints in new LOCK_DEBUG code. Jeff Janes http://git.postgresql.org/pg/commitdiff/bed756a820a2c1ee359f5f5b44806e3599190e95
  • Tweak __attribute__-wrapping macros for better pgindent results. This improves on commit bbfd7edae5aa5ad5553d3c7e102f2e450d4380d4 by making two simple changes: * pg_attribute_noreturn now takes parentheses, ie pg_attribute_noreturn(). Likewise pg_attribute_unused(), pg_attribute_packed(). This reduces pgindent's tendency to misformat declarations involving them. * attributes are now always attached to function declarations, not definitions. Previously some places were taking creative shortcuts, which were not merely candidates for bad misformatting by pgindent but often were outright wrong anyway. (It does little good to put a noreturn annotation where callers can't see it.) In any case, if we would like to believe that these macros can be used with non-gcc compilers, we should avoid gratuitous variance in usage patterns. I also went through and manually improved the formatting of a lot of declarations, and got rid of excessively repetitive (and now obsolete anyway) comments informing the reader what pg_attribute_printf is for. http://git.postgresql.org/pg/commitdiff/785941cdc359c6e595201ffb0df9d28f3f7173a4
  • Better fix for misuse of Float8GetDatumFast(). We can use that macro as long as we put the value into a local variable. Commit 735cd6128 was not wrong on its own terms, but I think this way looks nicer, and it should save a few cycles on 32-bit machines. http://git.postgresql.org/pg/commitdiff/2c33e0fbceb01d0ecd78330feef1315682c64bc4
  • Minor code cleanups in pgbench expression support. Get rid of unnecessary expr_yylex declaration (we haven't supported flex 2.5.4 in a long time, and even if we still did, the declaration in pgbench.h makes this one unnecessary and inappropriate). Fix copyright dates, improve some layout choices, etc. http://git.postgresql.org/pg/commitdiff/e9dd03c03aeed6486129e3101695b13d469c2985
  • Remove a couple other vestigial yylex() declarations. These were workarounds for a long-gone flex bug; all supported versions of flex emit an extern declaration as expected. http://git.postgresql.org/pg/commitdiff/9a8e23311cac14168df6644e03d533a4b07f933e
  • Make ginbuild's funcCtx be independent of its tmpCtx. Previously the funcCtx was a child of the tmpCtx, but that was broken by commit eaa5808e8ec4e82ce1a87103a6b6f687666e4e4c, which made MemoryContextReset() delete, not reset, child contexts. The behavior of having a tmpCtx reset also clear the other context seems rather dubious anyway, so let's just disentangle them. Per report from Erik Rijkers. In passing, fix badly-inaccurate comments about these contexts. http://git.postgresql.org/pg/commitdiff/1601830ec20d56dc7bf6b60a34f69841429e4825
  • Add vacuum_delay_point call in compute_index_stats's per-sample-row loop. Slow functions in index expressions might cause this loop to take long enough to make it worth being cancellable. Probably it would be enough to call CHECK_FOR_INTERRUPTS here, but for consistency with other per-sample-row loops in this file, let's use vacuum_delay_point. Report and patch by Jeff Janes. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/e4cbfd673d530a3e841db26a74f22e11a991205a

Bruce Momjian a poussé :

Kevin Grittner a poussé :

  • Reduce pinning and buffer content locking for btree scans. Even though the main benefit of the Lehman and Yao algorithm for btrees is that no locks need be held between page reads in an index search, we were holding a buffer pin on each leaf page after it was read until we were ready to read the next one. The reason was so that we could treat this as a weak lock to create an "interlock" with vacuum's deletion of heap line pointers, even though our README file pointed out that this was not necessary for a scan using an MVCC snapshot. The main goal of this patch is to reduce the blocking of vacuum processes by in-progress btree index scans (including a cursor which is idle), but the code rearrangement also allows for one less buffer content lock to be taken when a forward scan steps from one page to the next, which results in a small but consistent performance improvement in many workloads. This patch leaves behavior unchanged for some cases, which can be addressed separately so that each case can be evaluated on its own merits. These unchanged cases are when a scan uses a non-MVCC snapshot, an index-only scan, and a scan of a btree index for which modifications are not WAL-logged. If later patches allow all of these cases to drop the buffer pin after reading a leaf page, then the btree vacuum process can be simplified; it will no longer need the "super-exclusive" lock to delete tuples from a page. Reviewed by Heikki Linnakangas and Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/2ed5b87f96d473962ec5230fd820abfeaccb2069

Tatsuo Ishii a poussé :

Andrew Dunstan a poussé :

Peter Eisentraut a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Pavel Stehule sent in two more revisions of a patch to add a --strict option to pg_dump for missing tables.
  • Bruce Momjian sent in three more revisions of a patch to add the asciidoc output format to psql.
  • David Steele sent in two more revisions of a patch to to add a pg_audit extension.
  • Peter Geoghegan and Andrew (RhodiumToad) Gierth sent in two more revisions of a patch to add abbreviated keys for NUMERIC.
  • Michael Paquier sent in two revisions of a patch to expose PG_VERSION_NUM to developers.
  • Michael Paquier sent in another revision of a patch to add a table-level log_autovacuum_min_duration.
  • Kyotaro HORIGUCHI sent in a patch to remove individual phases to mark unique joins.
  • Andres Freund sent in another revision of a patch to add replication identifiers.
  • Amit Kapila sent in a patch to fix a bug in dsm_attach().
  • Peter Geoghegan sent in a patch to add DatumGetUInt32() around the hash_any() and hash_uint32() calls within varlena.c.
  • Heikki Linnakangas sent in another revision of a patch to use Intel SSE4.2 CRC instructions where available.
  • Álvaro Herrera sent in another revision of a patch to implement parsing utility commands.
  • Kaigai Kouhei sent in another revision of a patch to add a custom JOIN API.
  • Fabrízio de Royes Mello sent in a patch to fix a bug with indexes on unlogged tables.
  • Amit Kapila sent in two more revisions of a patch to implement parallel sequential scan.
  • Antonin Houska sent in two revisions of a WIP patch to enable spliting hash index buckets.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT IGNORE (and UPDATE).
  • Haribabu Kommi sent in another revision of a patch to add a catalog view to pg_hba.conf.
  • David Rowley sent in another revision of a patch to improve performance for outer joins where the outer side is unique.
  • Tom Lane sent in another revision of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Fabien COELHO sent in another revision of a patch to improve pgbench syntax error messages.
  • Fabien COELHO sent in another revision of a patch to extend pgbench expressions with functions.
  • Pavel Stehule sent in another revision of a patch to add a row_to_array function.
  • Michael Paquier sent in a patch to describe precise rounding behavior of numeric and double precision in the docs.
  • Andres Freund sent in a patch to make heap extension scale better.

par N Bougain le samedi 11 avril 2015 à 16h27

jeudi 2 avril 2015

Dimitri Fontaine

pgDay Paris

Le 21 avril prochain se tient le premier pgDay Paris: une conférence PostgreSQL d'une journée complète. Il s'agit de 8 conférences sur votre base de données préférée par des conférencers internationaux, incluant des retours d'expérience et une analyse de l'utilisation des derniers développements en cours dans notre projet de base de données préféré.

pgDay Paris, 21 avril 2015 : 8 conférences PostgreSQL

Pour 65 € l'accès à la conférence inclut les pauses cafés (avec thé, eau, biscuits, etc) et le déjeuner sur place (repas complet servi à table).

Demandez le programme !

Le programme de pgDay.paris est publié et sa lecture permet de positionner pgDay Paris comme une conférence PostgreSQL incontournable en France.

Nous commencerons avec une présentation de l'utilisation de PostgreSQL dans le groupe ADEO, dont vous connaissez certainement plusieurs des enseignes : les magazins Leroy Merlin, Brico Center ou encore weldom font partie du groupe. Leurs logiciels de caisse utilisent PostgreSQL afin de ne jamais perdre une transaction de paiement !

Ensuite Stéphane Schildknecht nous détaillera un cas d'usage intéressant de PostgreSQL avec de la réplication Slony. Puis Magnus Hagander, committer PostgreSQL et membre de la Core Team du projet, viendra nous présenter les nouveautés de PostgreSQL 9.5, qui va bientôt entrer en période de béta.

Le projet Open Source ToroDB nous sera ensuite présenté par son auteur principal, Álvaro Hernandez. Il s'agit d'une implémentation complète du protocole MongoDB qui utilise PostgreSQL afin de gérer ses données. Pour tout ceux qui se posent la question du NoSQL à comparer au relationnel, ToroDB apporte une réponse intéressante tant du point de vue de l'utilisation en tant que développeur que des performances (volumes stockés, temps d'insertion ou manipulation des données, temps d'exécution des requêtes).

Ensuite Grégoire Hubert viendra lui-même présenter POMM, une approche moderne d'intégration de la base de données et du code orienté objet. Cette approche permet de dépasser les limitations classiques des ORMs et d'utiliser l'ensemble des fonctionnalités du moteur de base de données utilisé !

Nous parlerons égalemment des nouveaux projets de réplication intégrés à PostgreSQL, BDR pour Bi-Directional Replication. Développeur au coeur du projet, Petr Jelinek interviendra afin de montrer comment les solutions BDR et UDR peuvent être utilisées pour résoudre des problématiques classiques de Haute Disponibilité des services et des données avec PostgreSQL.

Les Systèmes d'information géographique seront à l'honneur avec Vincent Picavet, contributeur à PostGIS, qui viendra nous expliquer ce que l'on peut faire de PostgreSQL avec les extension GIS.

Nous terminerons cette journée riche en conférences avec une présentation des progrès de PostgreSQL dans le passé et à l'avenir, ainsi qu'une analyse du positionnement de notre Système de Gestion de Données Relationnelles préféré. Simon Riggs contribue à PostgreSQL depuis plus de 10 ans et a développé des fonctionnalités essentielles : Point In Time Recovery, Hot Standy, Synchronous Replication sont quelques exemples. Il dirige l'effort de développement de l'avenir de la réplication dans PostgreSQL avec Logical Decoding en 9.4 et Bi-Directional Replication. Committer PostgreSQL très actif, DBA certifié Oracle® et Teradata, Simon est le conférencier idéal lorsqu'il s'agit de positionner PostgreSQL dans le contexte actuel.

Conférence

La conférence pgDay Paris aura lieu le 21 avril prochain au siège de La Poste (les détails et la carte sont sur http://pgday.paris/contact/), et nous vous attendons nombreux : Inscrivez-vous dès aujourd'hui !

par dim@tapoueh.org (Dimitri Fontaine) le jeudi 2 avril 2015 à 09h50

dimanche 29 mars 2015

Guillaume Lelarge

Version 0.2 de mon livre sur PostgreSQL

La version 0.2 de mon livre est sortie hier. En dehors des ajouts/correctifs dans les précédents chapitres, elle ajoute deux nouveaux chapitres.

Le premier concerne le protocole de communication client/serveur de PostgreSQL. Il permet de bien prendre conscience du dialogue et des possibilités d'échange entre ces deux entités. Le second aborde la question des connexions : comment s'établie une connexion, quels paramètres de configuration existent pour les connexions, comment gérer les connexions, etc.

De plus, le site d-booker, éditeur du livre, publie une interview de l'auteur (donc moi).

Si vous avez lu le livre « PostgreSQL, architectures et notions avancées », j'aimerais beaucoup savoir ce que vous en avez pensé. N'hésitez pas à intervenir sur le forum pour me remonter vos impressions ou tout problème que vous aurez constaté.

par Guillaume Lelarge le dimanche 29 mars 2015 à 16h46

mercredi 25 mars 2015

Rodolphe Quiédeville

PgDay Paris

J'ai la plaisir cette année de participer au comité de sélection du PgDay Paris qui se déroulera le 21 Avril 2015.

Le pgDay Paris est une journée de conférences et d'échanges organisée par la communauté française de PostgreSQL. Un ensemble de présentations en anglais et en français sera proposé, couvrant des sujets techniques ainsi que des retours d'expérience d'utilisation de PostgreSQL en production.

Que vous soyez développeur, administrateur système ou de bases de données ou bien « décideur » (DSI, directeur technique, etc), nous aurons du contenu pour vous !

Les inscriptions à l'événement sont dès maintenant disponibles pour le prix de 65 € pour la journée, qui inclut les pauses cafés et le déjeuner complet sur place.

Les places étant limitées, je vous invite à vous inscrire au plus tôt afin d'être sûr de pouvoir venir.

  • https://www.postgresql.eu/events/register/pgdayparis2015/

Le prix des places est maintenu volontairement bas afin de permettre au plus grand nombre de participer. Cela est rendu possible grâce au soutien des sponsors, et il reste là aussi des places. Alors si vous souhaitez apporter votre contribution au développement de PostgreSQL n'hésitez pas à prendre contact, toutes les coordonnées sont sur le site de l'évènement.

Rendez-vous le 21 !

par Rodolphe Quiédeville le mercredi 25 mars 2015 à 20h18

dimanche 22 mars 2015

Adrien Nayrat

Restauration PITR – Part 7

Je profite d’un week end de mauvais temps (m’empêchant de retourner en falaise :( ) pour poursuivre mes articles sur Postgres.

Dans le précédent article : Backup, restauration – Part 6 J’ai évoqué la restauration PITR pour Point In Time Recovery. Je vous encourage vivement à mettre en place ce qui va suivre afin de réduire la perte de données en cas de modification accidentelle sur des enregistrements : Un drop table par exemple.

Pour la mettre en place il vous faut :

  • Une instance Postgres maître.
  • Archivage des WAL, externalisé de préférence.

Je ne vais pas revenir sur la mise en place de l’archivage, je l’ai déjà détaillé dans les précédents articles. Pour vous expliquer cette restauration je vais utiliser un scénario “catastrophe”.

Il est 12H30 et on vous informe qu’une table a été supprimée par erreur sur un serveur de production.

La réplication que vous avez mis en place ne vous est d’aucune aide car l’opération a été reportée sur le réplicat.

Remarque : Avec la version 9.4 il est possible d’avoir un réplicat avec du retard. Avec un peu de chance l’opération de suppression de la table n’est peut être pas encore réalisée sur le réplicat.

Par chance, vous effectuez un backup full toutes les nuits à 1H00 et vous avez mis en place l’archivage des journaux. Leur nettoyage se fait après le backup full grâce à pg_archivecleanup.

Pour être tranquille vous effectuez un checkpoint afin que les derniers journaux soient archivés. Ensuite vous stoppez votre instance Postgres et déplacez le répertoire (au cas où).

Vous restaurez les fichiers du backup full. Il ne vous reste plus qu’a configurer correctement la restauration via le fichier recovery.conf.

Celui-ci devra contenir le paramètre restore_command afin que Postgres récupére les journaux de transactions archivés.

Il devra également contenir un paramètre pour la cible de récupération, Si on ne le spécifie pas le moteur va rejouer les journaux de transaction dans leur totalité, jusqu’au “drop table” fatal. Il est possible de demander la restauration jusqu’à un point de restauration, une date ou un identifiant de transaction.

Pour approfondir : http://docs.postgresqlfr.org/current/recovery-target-settings.html

Revenons à notre scénario, imaginons que nous souhaitons revenir à la dernière transaction jouée avant le drop fatal. En grattant un peu, on peut utiliser l’outil  pg_xlogdump pour lire les journaux de transaction.

/usr/lib/postgresql/9.3/bin/pg_xlogdump 000000010000000300000066
rmgr: Heap        len (rec/tot):     21/   197, tx:    1213547, lsn: 3/66000028, prev 3/650000F0, bkp: 1000, desc: insert: rel 1663/16486/16519; tid 0/2
rmgr: Transaction len (rec/tot):     12/    44, tx:    1213547, lsn: 3/660000F0, prev 3/66000028, bkp: 0000, desc: commit: 2015-03-22 12:23:35.834785 CET
rmgr: Standby     len (rec/tot):     16/    48, tx:    1213548, lsn: 3/66000120, prev 3/660000F0, bkp: 0000, desc: AccessExclusive locks: xid 1213548 db 16486 rel 16519
rmgr: Standby     len (rec/tot):     16/    48, tx:    1213548, lsn: 3/66000150, prev 3/66000120, bkp: 0000, desc: AccessExclusive locks: xid 1213548 db 16486 rel 16522
rmgr: Standby     len (rec/tot):     16/    48, tx:    1213548, lsn: 3/66000180, prev 3/66000150, bkp: 0000, desc: AccessExclusive locks: xid 1213548 db 16486 rel 16524
rmgr: Heap        len (rec/tot):     26/  6642, tx:    1213548, lsn: 3/660001B0, prev 3/66000180, bkp: 1000, desc: delete: rel 1663/16486/11774; tid 7/35 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  3270, tx:    1213548, lsn: 3/66001BA8, prev 3/660001B0, bkp: 1000, desc: delete: rel 1663/16486/11903; tid 46/44 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  4774, tx:    1213548, lsn: 3/66002888, prev 3/66001BA8, bkp: 1000, desc: delete: rel 1663/16486/11821; tid 2/26 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  5730, tx:    1213548, lsn: 3/66003B30, prev 3/66002888, bkp: 1000, desc: delete: rel 1663/16486/11786; tid 42/37 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/    58, tx:    1213548, lsn: 3/660051B0, prev 3/66003B30, bkp: 0000, desc: delete: rel 1663/16486/11786; tid 42/38 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  7902, tx:    1213548, lsn: 3/660051F0, prev 3/660051B0, bkp: 1000, desc: delete: rel 1663/16486/11797; tid 0/80 KEYS_UPDATED

C’est un extrait du résultat de la commande, le première ligne indique un insert avec la transaction N° 1213547, la ligne ne dessous indique un commit. En revanche à la troisième ligne on est passé à la transaction 1213548 avec un lock suivi de plusieurs lignes indiquant un delete. Il s’agit du drop fatal.

On va donc revenir à la transaction N°1213547. Voici le contenu de mon fichier recovery.conf :

restore_command = 'rsync "192.168.1.128:/var/lib/postgresql/9.3/wal_archive/%f" "%p"'

recovery_target_xid = '1213547'
recovery_target_inclusive = 'true'

La directive recovery_target_inclusive indique au moteur s’il inclue la transaction correspondant au N°1213547 ou s’il s’arrête juste avant. Ainsi j’aurais pu spécifier un xid de 1213548  et un target_inclusive à “false” pour obtenir le même résultat.

Ensuite il suffit de redémarrer postgres, voici ce qu’on peut observer dans les logs :

2015-03-22 12:41:17 CET LOG:  database system was interrupted; last known up at 2015-03-22 12:22:59 CET
2015-03-22 12:41:17 CET LOG:  creating missing WAL directory "pg_xlog/archive_status"
2015-03-22 12:41:17 CET LOG:  starting point-in-time recovery to XID 1213547
2015-03-22 12:41:17 CET LOG:  connection received: host=[local]
2015-03-22 12:41:17 CET LOG:  incomplete startup packet
2015-03-22 12:41:18 CET LOG:  connection received: host=[local]
2015-03-22 12:41:18 CET FATAL:  the database system is starting up
2015-03-22 12:41:18 CET LOG:  restored log file "000000010000000300000065" from archive
2015-03-22 12:41:18 CET LOG:  redo starts at 3/65000028
2015-03-22 12:41:18 CET LOG:  consistent recovery state reached at 3/650000F0
2015-03-22 12:41:18 CET LOG:  connection received: host=[local]
2015-03-22 12:41:18 CET FATAL:  the database system is starting up
2015-03-22 12:41:19 CET LOG:  connection received: host=[local]
2015-03-22 12:41:19 CET FATAL:  the database system is starting up
2015-03-22 12:41:19 CET LOG:  restored log file "000000010000000300000066" from archive
2015-03-22 12:41:19 CET LOG:  recovery stopping after commit of transaction 1213547, time 2015-03-22 12:23:35.834785+01
2015-03-22 12:41:19 CET LOG:  redo done at 3/660000F0
2015-03-22 12:41:19 CET LOG:  last completed transaction was at log time 2015-03-22 12:23:35.834785+01
rsync: link_stat "/var/lib/postgresql/9.3/wal_archive/00000002.history" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [Receiver=3.0.9]
2015-03-22 12:41:19 CET LOG:  selected new timeline ID: 2
rsync: link_stat "/var/lib/postgresql/9.3/wal_archive/00000001.history" failed: No such file or directory (2)
2015-03-22 12:41:19 CET LOG:  connection received: host=[local]
2015-03-22 12:41:19 CET FATAL:  the database system is starting up
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [Receiver=3.0.9]

On remarque que le moteur créée le répertoire des journaux de transaction, ensuite la ligne suivante est assez explicite : “starting point-in-time recovery to XID 1213547″. Quelques lignes plus bas on remarque qu’il rejoue les journaux de transaction (“restored log file “000000010000000300000065” from archive”).

Il arrête la restauration jusqu’à la transaction spécifiée :

recovery stopping after commit of transaction 1213547, time 2015-03-22 12:23:35.834785+01

On remarque quelques lignes d’erreur correspondant à la sortie de rsync. Visiblement il cherche à copier plusieurs fichiers en “x.history”.  En regardant dans le répertoire d’archivage on constate que le moteur a créé un fichier 00000002.history contenant :

1       3/66000120      after transaction 1213547

Point important également, on constate que la numérotation des journaux de transaction a changée :

-rw------- 1 postgres postgres 16777216 mars  22 12:23 000000010000000300000065
-rw------- 1 postgres postgres      305 mars  22 12:23 000000010000000300000065.00000028.backup
-rw------- 1 postgres postgres 16777216 mars  22 12:41 000000010000000300000066
-rw------- 1 postgres postgres 16777216 mars  22 12:46 000000020000000300000066
-rw------- 1 postgres postgres       39 mars  22 12:41 00000002.history

Le fichier “000000010000000300000065” correspond au wal lors du pg_basebackup, “000000010000000300000066” correspond au checkpoint que j’ai lancé après l’incident. Après la restauration on peut voir le fichier “000000020000000300000066”. Le huitième caractère est passé de 1 à 2.

En retournant dans les logs du moteur on peut lire la ligne suivante un peu plus bas : “selected new timeline ID: 2″En fait, lors de la restauration, Postgres créée une nouvelle timeline. C’est un concept que j’aborderai plus tard, en attendant je vous invite à voir cette présentation sur le fonctionnement des timeline après une restauration PIRT : https://wiki.postgresql.org/images/e/e5/FOSDEM2013-Timelines.pdf

Bien entendu comme tout système de sauvegarde vous devez jouer la procédure afin de valider qu’il y a pas eu d’erreur ou oubli.

 

FacebookTwitterGoogle+ViadeoPrintEmailGoogle GmailLinkedInPartager

par Adrien Nayrat le dimanche 22 mars 2015 à 12h25

lundi 16 mars 2015

Damien Clochard

70 Nuances de Postgres

Le support du standard SQL/MED a été introduit dans PostgreSQL 9.1 (2011). Quatre ans plus tard, nous avons maintenant plus de 70 Foreign Data Wrappers (FDW) disponibles pour lire et écrire sur tout type de moteur de stockage: base Oracle ou MongoDB, fichiers XML, cluster Hadoop, dépot Git, Twitter, etc. etc. etc.

Il y a quelque jours, pendant le FOSDEM j’ai assisté à la conférence de mon collègue Ronan Dunklau intitulée Foreign Data Wrappers in PostgreSQL : Where are we now ? Ronan est l’auteur de l’extension multicorn et pendant sa présentation je me suis rappelé à quel point multicorn est probablement le projet le plus sous-estimé de la communauté PostgreSQL. En guise d’exercice, j’ai commencé à compter tous les FDW que je connaissais… et j’ai rapidement réalisé qu’il y en avait trop pour ma mémoire de poisson rouge !

Du coup, je suis retourné sur la liste de FDW du wiki PostgreSQL. J’ai commencé à mettre à jour et reformatter le catalogue de tous les wrappers existants et j’ai abouti à une liste de plus de 70 wrappers pour PostgreSQL…

Quelques leçons à en tirer

Pendant cet inventaire, j’ai découvert quelques points intéressants:

  • Tous les SGBD majeurs sont couverts à l’exception de DB2. Bizarrement DB2 est aussi le seul SGBD majeur ayant une implémentation du standard SQL/MED. Difficile de dire si ces deux faits sont liés ou pas…

  • Un tiers des FDW sont écrits en Python et basés sur Multicorn. Les autres sont des wrappers “natifs” écrits en C. Assez logiquement les wrappers en C sont prvilégiés pour les connecteurs de SGBD (ODBC, Oracle, etc.) pour des raisons de performance. En parallèle Multicorn est utilisé majoritairement pour les interroger des web services (S3 storage,RSS files, etc.) et des formats de données specifiques (fichiers de genotype, GeoJSON, etc.)

  • Je ne pense pas que c’était le but initial mais les Foreign Data Wrappers sont aussi un vecteur d’innovation. Certains connecteurs sont détournés de leur usage premier pour développer de nouvelles fonctionnalités. Par exemple, CitusDB a publié une base de données orientée colonnes basée sur un FDW, d’autres wrappers vont dans des directions encore plus exotiques comme la collecte de stats au niveau OS, de l’accéleration GPU pour les seq scans ou un moteur traitement parallèle de requêtes… La plupart de ces projets ne conviennent pas pour de la production bien sur. Cependant cela démontre qu’il est aisé d’implémenter un PoC et à quel point cette technologie est versatile.

  • Le réseau de distribution PGXN semble plutot méconnu. Seulement 20% des wrappers sont disponibles via PGXN. Ou peut-être que les développeurs sont trop paresseux pour créer un paquet pour leur extension ?

  • Le succès de l’implémentation d’ SQL/MED dans PostgreSQL a toutefois des inconvénients. A vue de nez, il y aura plus de 100 wrappers disponibles à la fin 2015. Pour certaines sources de données comme mongodb ou LDAP, il y a déjà plusieurs wrappers différents, sans parler de la cohorte de connecteurs Hadoop :) Sur le long terme, les utilisateurs de PostgreSQL auront du mal à déterminer quels sont les wrappers maintenus et lesquels sont abandonnés… La page de wiki est une tentative pour répondre à cette question mais il y a surement d’autres solutions pour fournir des informations précises et exhaustives aux utilisateurs. Peut-être qu’il faudrait un data Wrapper qui pourrait lister tous les data wrappers :)

The next big thing : IMPORT FOREIGN SCHEMA

De plus, il reste une limitation majeure à l’implémentation actuelle : les meta données. Actuellement, il n’y a pas de moyen simple d’utiliser les (éventuelles) capacités d’introspection de la source de données distante. En clair : vous devez créer toutes vos tables externes une par une. Lorsque l’on veut se connecter à toutes les tables d’une base distantes, c’est répétitif et il y a un risque d’erreurs. Si vous vous connectez à une base PostgreSQL distante, vous pouvez utiliser des astuces mais clairement c’est une tache qui devrait être faite au niveau du connecteur lui-même.

Heureusement voici la commande IMPORT FOREIGN SCHEMA ! Cette nouvelle fonctionnalité a été écrite par Ronan Dunklau, Michael Paquier et Tom Lane. Vous pouvez lire une démo rapide sur le blog de Michael. Cette commande sera disponible dans la version 9.5.

C’est une amélioration énorme ! Avec cet IMPORT et les douzaines de wrappers disponibles, PostgreSQL est en passe de devenir une plateforme d’intégration de données unique et le besoin d’utiliser des outils d’ETL externes devient de plus en plus faible.

PostgreSQL as a Data Integration Plateform

Liens :

EDIT : Merci à Eric Pommereau et Guénolé Marquier pour la relecture !

lundi 16 mars 2015 à 11h17

jeudi 12 mars 2015

Nicolas Thauvin

pg_back le script de base pour sauvegarder PostgreSQL

Il y a fort longtemps, et c'est ma première contribution relative à PostgreSQL, j'ai écrit un script de backup qui dump tout un serveur PostgreSQL avec pg_dump et pg_dumpall. Il s'agit de pg_back.

Cela peut paraître curieux de publier un simple script de sauvegarde que tout DBA PostgreSQL a écrit dans sa vie et sait écrire par cœur. Surtout qu'on le réécrit en permanence ce script, pour ajuster des chemins, des cas particuliers du serveur à sauvegarder et de l'environnement où l'on sauvegarde...

En bien justement, c'est parce qu'on le réécrit tout le temps que pg_back fait gagner du temps. Il est simple et court, facilement lisible, c'est du shell : tout ce qu'il faut pour en faire une bonne base pour créer un script de sauvegarde adapté. Quand on l'utilise comme patron pour en faire un outil plus évolué, on gagne du temps.

Justement rajouter du code pour l'adapter peut se faire au début. Si on n'a pas envie d'utiliser le fichier de configuration, on adapte la liste de variables au début du script, quitte à en rajouter.

L'autre endroit intéressant c'est tout à la fin, avant le exit, on peut rajouter tout ce qu'il faut pour externaliser ses sauvegardes.

C'est par ici.

jeudi 12 mars 2015 à 15h24

dimanche 22 février 2015

Guillaume Lelarge

Durée d'établissement d'une connexion

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

J'ai toujours eu en tête qu'une connexion mettait du temps à s'établir entre un client et PostgreSQL. J'avais en tête un nombre qui me semblait plausible mais j'avoue que je n'avais jamais fait réellement le test.

Ce week-end, travaillant sur le chapitre sur la gestion des connexions, je me suis demandé si on pouvait calculer ce temps. J'ai donc regardé le code des processus postmaster/postgres pour ajouter quelques traces, histoire d'en savoir plus. Voici le patch que j'ai réalisé :

diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index f05114d..9d8fb8a 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2198,6 +2198,8 @@ ConnCreate(int serverFd)
 {
        Port       *port;
 
+       elog(LOG, "patch - ConnCreate(%d)", serverFd);
+
        if (!(port = (Port *) calloc(1, sizeof(Port))))
        {
                ereport(LOG,
@@ -3760,6 +3762,8 @@ BackendStartup(Port *port)
        Backend    *bn;                         /* for backend cleanup */
        pid_t           pid;
 
+       elog(LOG, "patch - BackendStart()");
+
        /*
         * Create backend data structure.  Better before the fork() so we can
         * handle failure cleanly.
@@ -3814,6 +3818,8 @@ BackendStartup(Port *port)
 
                MyProcPid = getpid();   /* reset MyProcPid */
 
+               elog(LOG, "patch - new pid is %d", MyProcPid);
+
                MyStartTime = time(NULL);
 
                /* We don't want the postmaster's proc_exit() handlers */
@@ -3916,6 +3922,8 @@ BackendInitialize(Port *port)
        char            remote_port[NI_MAXSERV];
        char            remote_ps_data[NI_MAXHOST];
 
+       elog(LOG, "patch - BackendInitialize()");
+
        /* Save port etc. for ps status */
        MyProcPort = port;
 
@@ -4096,6 +4104,8 @@ BackendRun(Port *port)
        int                     usecs;
        int                     i;
 
+       elog(LOG, "patch - BackendRun()");
+
        /*
         * Don't want backend to be able to see the postmaster random number
         * generator state.  We have to clobber the static random_seed *and* start
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index bc4eb33..4e1a3f7 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3578,6 +3578,7 @@ PostgresMain(int argc, char *argv[],
        sigjmp_buf      local_sigjmp_buf;
        volatile bool send_ready_for_query = true;
 
+       elog(LOG, "patch - PostgresMain()");
        /*
         * Initialize globals (already done if under postmaster, but not if
         * standalone).
@@ -3845,6 +3846,8 @@ PostgresMain(int argc, char *argv[],
         * were inside a transaction.
         */
 
+       elog(LOG, "patch - PostgresMain() - ready to execute command");
+
        if (sigsetjmp(local_sigjmp_buf, 1) != 0)
        {
                /*
@@ -4056,12 +4059,16 @@ PostgresMain(int argc, char *argv[],
                if (ignore_till_sync && firstchar != EOF)
                        continue;
 
+               elog(LOG, "patch - PostgresMain() - processing command");
+
                switch (firstchar)
                {
                        case 'Q':                       /* simple query */
                                {
                                        const char *query_string;
 
+                                       elog(LOG, "patch - PostgresMain() - executing simple query");
+
                                        /* Set statement_timestamp() */
                                        SetCurrentStatementStartTimestamp();
 
@@ -4279,6 +4286,8 @@ PostgresMain(int argc, char *argv[],
                        case 'X':
                        case EOF:
 
+                               elog(LOG, "patch - PostgresMain() - exiting");
+
                                /*
                                 * Reset whereToSendOutput to prevent ereport from attempting
                                 * to send any more messages to client.

En configurant PostgreSQL pour qu'il ajoute la date (à la milliseconde près) et le PID, et en configurant la trace des connexions et déconnexions :

log_min_duration_statement = 0
log_connections = on
log_disconnections = on
log_line_prefix = '%m [%p] '

et en exécutant la commande suivante :

$ psql -c "select * from t1 limit 200" b1

nous obtenons les traces suivantes :

2015-02-22 22:47:23.022 CET [6087] LOG:  patch - ConnCreate(5)
2015-02-22 22:47:23.022 CET [6087] LOG:  patch - BackendStart()
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - new pid is 6283
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - BackendInitialize()
2015-02-22 22:47:23.023 CET [6283] LOG:  connection received: host=[local]
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - BackendRun()
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - PostgresMain()
2015-02-22 22:47:23.025 CET [6283] LOG:  connection authorized: user=postgres database=b1
2015-02-22 22:47:23.027 CET [6283] LOG:  patch - PostgresMain() - ready to execute command
2015-02-22 22:47:23.027 CET [6283] LOG:  patch - PostgresMain() - processing command
2015-02-22 22:47:23.028 CET [6283] LOG:  patch - PostgresMain() - executing simple query
2015-02-22 22:47:23.028 CET [6283] LOG:  duration: 0.691 ms  statement: select * from t1 limit 200
2015-02-22 22:47:23.736 CET [6283] LOG:  patch - PostgresMain() - processing command
2015-02-22 22:47:23.736 CET [6283] LOG:  patch - PostgresMain() - exiting
2015-02-22 22:47:23.737 CET [6283] LOG:  disconnection: session time: 0:00:00.913 user=postgres database=b1 host=[local]

Autrement dit, il faut compter quelques millisecondes pour établir une connexion sans pooler. Après différents tests (impliquant notamment pgbench), le pire que j'ai vu est 10 millisecondes. Pas bien méchant quand on y pense. J'ai aussi noté que la toute première connexion était bien plus lente (dans les 40 millisecondes), ce qui reste encore bien loin de ce que j'imaginais.

J'ai aussi testé avec différentes valeurs du shared_buffers car il semblerait que la taille mémoire d'un processus a une importance dans la durée d'exécution de l'appel système fork().

Comme quoi il est vraiment préférable de tout tester pour ne pas avoir d'idées préconçues.

par Guillaume Lelarge le dimanche 22 février 2015 à 21h49

jeudi 19 février 2015

Guillaume Lelarge

PostgreSQL et la mémoire partagée

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

Lors de l'écriture du chapitre sur la gestion de la mémoire par PostgreSQL, j'ai cherché à différencier les différents blocs alloués en mémoire partagée. La documentation de PostgreSQL est assez peu bavard sur ce sujet, je me suis donc retourné vers le code source. Ce dernier donne énormément d'informations à qui se donne un peu la peine de les chercher. J'ai fini par trouver le fichier src/backend/storage/ipc/shmem.c qui s'occupe de la gestion de la mémoire partagée (shmem pour SHared MEMory).

Ce fichier contient différentes fonctions, dont la fonction ShmemInitStruct, dont le but est d'initialiser une structure en mémoire partagée. Il suffit de lui fournir le nom de la structure et sa taille, et la fonction se charge de l'allocation. Une petite modification de cette fonction permet d'afficher quelques informations à chaque appel de cette fonction. Voici ce patch :

diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 2ea2216..8703f48 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -402,6 +402,8 @@ ShmemInitStruct(const char *name, Size size, bool *foundPtr)
 	{
 		/* It isn't in the table yet. allocate and initialize it */
 		structPtr = ShmemAlloc(size);
+		ereport(LOG, (errmsg("allocate shared memory segment "%s", size %d",
+				name, (int)size)));
 		if (structPtr == NULL)
 		{
 			/* out of memory; remove the failed ShmemIndex entry */

Au lancement de PostgreSQL, modifié avec ce patch, nous obtenons cette sortie dans les traces :

LOG:  allocate shared memory segment "Control File", size 248 
LOG:  allocate shared memory segment "XLOG Ctl", size 16804496
LOG:  allocate shared memory segment "CLOG Ctl", size 525312
LOG:  allocate shared memory segment "SUBTRANS Ctl", size 263168
LOG:  allocate shared memory segment "MultiXactOffset Ctl", size 65856
LOG:  allocate shared memory segment "MultiXactMember Ctl", size 131648
LOG:  allocate shared memory segment "Shared MultiXact State", size 176 
LOG:  allocate shared memory segment "Buffer Descriptors", size 33554432
LOG:  allocate shared memory segment "Buffer Blocks", size 0
LOG:  allocate shared memory segment "Shared Buffer Lookup Table", size 32880
LOG:  allocate shared memory segment "Buffer Strategy Status", size 32
LOG:  allocate shared memory segment "LOCK hash", size 2160
LOG:  allocate shared memory segment "PROCLOCK hash", size 2160
LOG:  allocate shared memory segment "Fast Path Strong Relation Lock Data", size 4100
LOG:  allocate shared memory segment "PREDICATELOCKTARGET hash", size 2160
LOG:  allocate shared memory segment "PREDICATELOCK hash", size 2160
LOG:  allocate shared memory segment "PredXactList", size 88
LOG:  allocate shared memory segment "SERIALIZABLEXID hash", size 2160
LOG:  allocate shared memory segment "RWConflictPool", size 24
LOG:  allocate shared memory segment "FinishedSerializableTransactions", size 16
LOG:  allocate shared memory segment "OldSerXid SLRU Ctl", size 131648
LOG:  allocate shared memory segment "OldSerXidControlData", size 16
LOG:  allocate shared memory segment "Proc Header", size 88
LOG:  allocate shared memory segment "Proc Array", size 108 
LOG:  allocate shared memory segment "Backend Status Array", size 3672
LOG:  allocate shared memory segment "Backend Application Name Buffer", size 1088
LOG:  allocate shared memory segment "Backend Client Host Name Buffer", size 1088
LOG:  allocate shared memory segment "Backend Activity Buffer", size 17408
LOG:  allocate shared memory segment "Prepared Transaction Table", size 16
LOG:  allocate shared memory segment "Background Worker Data", size 1992
LOG:  allocate shared memory segment "shmInvalBuffer", size 66104
LOG:  allocate shared memory segment "PMSignalState", size 180 
LOG:  allocate shared memory segment "ProcSignalSlots", size 864 
LOG:  allocate shared memory segment "Checkpointer Data", size 10485800
LOG:  allocate shared memory segment "AutoVacuum Data", size 224 
LOG:  allocate shared memory segment "Wal Sender Ctl", size 56
LOG:  allocate shared memory segment "Wal Receiver Ctl", size 1192
LOG:  allocate shared memory segment "BTree Vacuum State", size 216 
LOG:  allocate shared memory segment "Sync Scan Locations List", size 656 
LOG:  allocate shared memory segment "Async Queue Control", size 244 
LOG:  allocate shared memory segment "Async Ctl", size 65856
LOG:  allocate shared memory segment "pg_stat_statements", size 48
LOG:  allocate shared memory segment "pg_stat_statements hash", size 2160

(Et comme on peut le constater par les deux dernières lignes, ce serveur avait l'extension pg_stat_statements à charger)

Nous connaissons ainsi les différents segments et leur taille respective. À partir de là, il a suffit de suivre la piste des différents segments pour comprendre leur utilité et la façon dont ils sont utilisés.

par Guillaume Lelarge le jeudi 19 février 2015 à 22h53

mercredi 21 janvier 2015

Guillaume Lelarge

PostgreSQL - Architecture et notions avancées

Et voilà, j'ai fini par le faire. J'ai fini par me convaincre d'écrire un livre complet sur PostgreSQL.

C'est quelque chose qui me trottait dans la tête depuis longtemps. Surtout depuis que Thomas Reiss m'avait montré le Concept Guide d'Oracle. Je m'étais dit à ce moment-là : mince, c'est ça qu'il manque à PostgreSQL. Mais bon, ça demande beaucoup de temps, beaucoup de travail. Je n'avais pas très envie de me jeter là-dedans, même si je savais que certains éditeurs cherchaient des auteurs pour des livres sur PostgreSQL.

J'ai eu la chance de connaître Patricia Montcorgé lors de la traduction du livre de Greg Smith sur les performances avec PostgreSQL. Si bien que, après avoir lu le livre de Brendan Gregg sur la recherche de performances (Systems Performance: Enterprise and the Cloud, excellent livre, à mettre entre toutes les mains), je lui ai proposé deux projets : la traduction de ce livre et l'écriture d'un livre sur PostgreSQL. Elle m'a appris qu'elle avait fondé sa propre maison d'édition, qu'elle ne s'occupait plus de traductions, et que le livre sur PostgreSQL pourrait l'intéresser. On s'est rencontré, et, après lui avoir expliqué plus en profondeur mon projet, elle a pu m'expliquer comment elle voyait le travail avec elle. J'ai trouvé que ça ressemblait beaucoup à un projet libre, avec une version beta, du git, du docbook, des mises à jour facile, etc. Bref, j'étais bien accroché.

Je travaille sur ce livre depuis avril 2014. On a déjà subi un gros retard à cause du chapitre sur les processus : étudier le code de chaque processus a été particulièrement long. Mais bon, on a maintenant un rythme de croisière intéressant. On s'est décidé sur un chapitre tous les 4 à 6 semaines.

La version beta est sortie aujourd'hui. Elle est disponible par module ou complète, uniquement en version électronique pour le moment (jusqu'à la version finale en fait). Les chapitres seront livrés au fur et à mesure de leur écriture. Un forum est disponible pour les lecteurs qui voudraient laisser des commentaires ou des demandes ou des corrections.

La version finale sera mise à jour pour intégrer les changements effectués par la version 9.5.

Bon, je retourne bosser sur le livre :)

Preneur de tout commentaire par mail sur guillaume@lelarge.info.

par Guillaume Lelarge le mercredi 21 janvier 2015 à 22h50

dimanche 28 décembre 2014

Guillaume Lelarge

Manuel de la 9.4 et articles GLMF

Trois billets en 2014 (avec celui-là), j'aurais fait fort :)

Néanmoins, quelques informations importantes.

Tout d'abord, la documentation de PostgreSQL 9.4 est enfin disponible entièrement en français, avec les différents formats habituels. C'est comme d'habitude sur http://docs.postgresql.fr.

Ensuite, au niveau des articles pour le GNU/Linux Magazine France, deux articles sont parus sur la version 9.4, un peu avant la sortie (les numéros 175 et 177). La suite de la série sur le planificateur est prévue rapidement avec un article sur les jointures. Il s'agit donc du deuxième article. Les troisième et quatrième articles sont déjà dans le bureau du rédacteur en chef (depuis août en fait). Ils devraient sortir rapidement après, suivant la place disponible dans le magazine.

Et voilà. L'année 2015 devrait être très intéressante :) Quelques infos plus tard à ce sujet.

par Guillaume Lelarge le dimanche 28 décembre 2014 à 11h36

lundi 22 septembre 2014

Nicolas Thauvin

Confiner PostgreSQL avec SELinux

J'avais déjà expérimenté un peu avec SELinux il y a deux ans sans aller trop loin, parce que j'entendais souvent la phrase "Si on veut de la sécurité, il faut SELinux" et surtout à cause de l'arrivée de l'extension sepgsql dans les modules contrib de PostgreSQL. Ça avait donné une conf pour le Fosdem où finalement, j'ai plus parlé des privilèges classiques que de SELinux.

Suite à une demande au ${BOULOT}, je me suis replongé dedans, et les choses ont peu évolué. Dans la majorité des recherches que j'ai pu faire, SELinux reste tout de même un truc qui se met en travers la route, c'est-à-dire, qu'il y a toujours plus de résultats sur comment le désactiver plutôt que configurer les choses correctement. Ensuite, certains proposent de faire aveuglement confiance à setroubleshoot et audit2allow pour faire taire la bête. Enfin, j'ai dû trouver une page ou deux après en deux semaines à temps plein sur le sujet qui parlent de comment confiner un utilisateur dans le but d'implémenter ce que promet SELinux et que toutes les entreprises demandent : le RBAC, Role Based Access Control, ou comment donner le moins de droits possibles aux sous-traitants qui hébergent les serveurs.

Cette fois ci, je suis allé plus loin. Déjà, je suis parti sur une installation de CentOS 6, la famille de distribution RHEL/CentOS/Fedora semble être la plus en avance par rapport à SELinux : à peu près tous les services/daemons fournis dans l'install sont confinés par SELinux. Les utilisateurs ne le sont pas au login, il n'y a donc aucune configuration de RBAC. On verra peut-être ça plus tard, j'ai pas encore obtenu de résultat satisfaisant sur ce sujet, et il vaut mieux expliquer comment confiner correctement PostgreSQL avant de passer aux roles. Tout simplement, parce qu'il est très simple de casser ce confinement avec la configuration par défaut de RHEL, grâce à pg_ctl. Enfin, l'installation des paquets du PGDG n'est pas confinée par défaut, il manque les file contexts adaptés aux chemins particuliers de ces paquets, prévu pour faire cohabiter plusieurs versions majeures ; ce qui ne correspond pas ce qu'à prévu Red Hat pour PostgreSQL.

Après cette longue introduction, mais avant de commencer, il faut savoir administrer un minumum SELinux : si vous ne savez pas qu'il existe des options -Z, ce que sont les contexts, les types et les domaines, mieux vaut d'abord se documenter, par exemple chez Red Hat.

Voici donc les file contexts à ajouter dans un fichier module.te pour confiner l'installation PostgreSQL du PGDG :

/etc/rc\.d/init\.d/(se)?postgresql(-.*)?    --  gen_context(system_u:object_r:postgresql_initrc_exec_t,s0)
/usr/pgsql-[0-9]+\.[0-9]+/bin/initdb        --  gen_context(system_u:object_r:postgresql_exec_t,s0)
/usr/pgsql-[0-9]+\.[0-9]+/bin/pg_ctl        --  gen_context(system_u:object_r:postgresql_initrc_exec_t,s0)
/usr/pgsql-[0-9]+\.[0-9]+/bin/postgres      --  gen_context(system_u:object_r:postgresql_exec_t,s0)
/usr/pgsql-[0-9]+.[0-9]+/share/locale(/.*)?     gen_context(system_u:object_r:locale_t,s0)
/usr/pgsql-[0-9]+.[0-9]+/share/man(/.*)?        gen_context(system_u:object_r:man_t,s0)

Tout d'abord, pour l'init script et pg_ctl, on utilise le type postgresql_initrc_exec_t, c'est ce qui permet de lancer PostgreSQL dans le domaine confiné postgresql_t au boot, via l'init script, et manuellement. La méthode la plus propre est de toujours utiliser l'init script, idéalement par l'intermédiaire de run_init pour démarrer, arrêter ou redémarrer le postmaster. On évite alors de laisser trainer des choses dans /var/{run,lock}.

Les programmes postgres et initdb doivent avoir le type postgresql_exec_t car ils exécutent le serveur PostgreSQL ; cela doit se faire dans le domaine postgresql_t.

Enfin, on a placé les labels corrects sur les fichiers de traduction et les pages de man, pour faire plus propre. Ce code source de module SELinux alors doit être compilé et chargé.

Cette configuration est reprise dans le module SELinux disponible sur github. On peut aussi l'ajouter manuellement avec semanage :

semanage fcontext -a -t postgresql_initrc_exec_t '/etc/rc\.d/init\.d/(se)?postgresql(-.*)?'
semanage fcontext -a -t postgresql_exec_t '/usr/pgsql-[0-9]+\.[0-9]+/bin/initdb'
semanage fcontext -a -t postgresql_initrc_exec_t '/usr/pgsql-[0-9]+\.[0-9]+/bin/pg_ctl'
semanage fcontext -a -t postgresql_exec_t '/usr/pgsql-[0-9]+\.[0-9]+/bin/postgres'
semanage fcontext -a -t locale_t '/usr/pgsql-[0-9]+.[0-9]+/share/locale(/.*)?'
semanage fcontext -a -t man_t '/usr/pgsql-[0-9]+.[0-9]+/share/man(/.*)?'

Il existe un certain nombre de booleans pour la configuration des droits SELinux de PostgreSQL, le plus important concerne rsync, souvent nécessaire pour faire des base backups. Il s'agit de postgresql_can_rsync, pour l'activer :

semanage boolean -m --on postgresql_can_rsync

Si on lance l'instance sur un port TCP différent de 5432, il faut l'autoriser dans la configuration locale de SELinux :

semanage port -a -t postgresql_port_t -p tcp <port>

Enfin, il ne faut pas oublier d'appliquer les contexts aux fichiers soit avec restorecon, un relabel complet au reboot ou chcon.

lundi 22 septembre 2014 à 14h23

Rodolphe Quiédeville

Django Paginator oui mais

Après la lecture de l'excellent article de Markus Winnand no-offset j'ai réalisé un test pour mesurer la différence de performance de l'utilisation d'OFFSET avec Django.

La suite à lire sur le blog Novapost's paradize.

par Rodolphe Quiédeville le lundi 22 septembre 2014 à 08h13

samedi 12 avril 2014

Guillaume Lelarge

Nouvelle série d'articles pour GLMF

Après l'article sur les nouveautés de la version 9.3, j'ai entamé une série d'articles sur le fonctionnement du planificateur (aussi appelé optimiseur).

Le premier est sorti ce mois-ci, dans le numéro 170 de GNU/Linux Magazine France. Il traite des différents types de parcours que l'optimiseur peut planifier.

J'ai loupé le coche pour le prochain numéro, donc celui sur les jointures devrait sortir dans le numéro 172 si tout va bien. En fait, je viens tout juste de finir son écriture.

Il y aura certainement deux autres articles, un sur les autres types de nœuds et un sur les outils pour la compréhension des plans d'exécution mais ils restent à écrire.

En attendant, je suis preneur de toute remarque/critique sur mes articles :)

par Guillaume Lelarge le samedi 12 avril 2014 à 16h11