PostgreSQL La base de donnees la plus sophistiquee au monde.

La planète francophone de PostgreSQL

lundi 29 juin 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 28 juin 2015

La PGConf.DE se tiendra à Hamburg (Allemagne) les 26 et 27 novembre au Lindner Hotel am Michel. L'appel à conférenciers court jusqu'au 13 septembre 2015 : http://2015.pgconf.de/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juin

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150628215355.GB30019@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Noah Misch pushed:

  • Truncate strings in tarCreateHeader() with strlcpy(), not sprintf(). This supplements the GNU libc bug #6530 workarounds introduced in commit 54cd4f04576833abc394e131288bf3dd7dcf4806. On affected systems, a tar-format pg_basebackup failed when some filename beneath the data directory was not valid character data in the postmaster/walsender locale. Back-patch to 9.1, where pg_basebackup was introduced. Extant, bug-prone conversion specifications receive only ASCII bytes or involve low-importance messages. http://git.postgresql.org/pg/commitdiff/4318118edd5582696027f357771e0a8b091fe2bf

Robert Haas pushed:

Tom Lane pushed:

  • Improve inheritance_planner()'s performance for large inheritance sets. Commit c03ad5602f529787968fa3201b35c119bbc6d782 introduced a planner performance regression for UPDATE/DELETE on large inheritance sets. It required copying the append_rel_list (which is of size proportional to the number of inherited tables) once for each inherited table, thus resulting in O(N^2) time and memory consumption. While it's difficult to avoid that in general, the extra work only has to be done for append_rel_list entries that actually reference subquery RTEs, which inheritance-set entries will not. So we can buy back essentially all of the loss in cases without subqueries in FROM; and even for those, the added work is mainly proportional to the number of UNION ALL subqueries. Back-patch to 9.2, like the previous commit. Tom Lane and Dean Rasheed, per a complaint from Thomas Munro. http://git.postgresql.org/pg/commitdiff/2cb9ec1bcb35dd6b4cf7a4a325aaa9791444e69d
  • Docs: fix claim that to_char('FM') removes trailing zeroes. Of course, what it removes is leading zeroes. Seems to have been a thinko in commit ffe92d15d53625d5ae0c23f4e1984ed43614a33d. Noted by Hubert Depesz Lubaczewski. http://git.postgresql.org/pg/commitdiff/d759b7eb6aee12bd52516905d790072845b4356f
  • Fix the logic for putting relations into the relcache init file. Commit f3b5565dd4e59576be4c772da364704863e6a835 was a couple of bricks shy of a load; specifically, it missed putting pg_trigger_tgrelid_tgname_index into the relcache init file, because that index is not used by any syscache. However, we have historically nailed that index into cache for performance reasons. The upshot was that load_relcache_init_file always decided that the init file was busted and silently ignored it, resulting in a significant hit to backend startup speed. To fix, reinstantiate RelationIdIsInInitFile() as a wrapper around RelationSupportsSysCache(), which can know about additional relations that should be in the init file despite being unknown to syscache.c. Also install some guards against future mistakes of this type: make write_relcache_init_file Assert that all nailed relations get written to the init file, and make load_relcache_init_file emit a WARNING if it takes the "wrong number of nailed relations" exit path. Now that we remove the init files during postmaster startup, that case should never occur in the field, even if we are starting a minor-version update that added or removed rels from the nailed set. So the warning shouldn't ever be seen by end users, but it will show up in the regression tests if somebody breaks this logic. Back-patch to all supported branches, like the previous commit. http://git.postgresql.org/pg/commitdiff/5d1ff6bd559ea8df1b7302e245e690b01b9a4fa4
  • Avoid passing NULL to memcmp() in lookups of zero-argument functions. A few places assumed they could pass NULL for the argtypes array when looking up functions known to have zero arguments. At first glance it seems that this should be safe enough, since memcmp() is surely not allowed to fetch any bytes if its count argument is zero. However, close reading of the C standard says that such calls have undefined behavior, so we'd probably best avoid it. Since the number of places doing this is quite small, and some other places looking up zero-argument functions were already passing dummy arrays, let's standardize on the latter solution rather than hacking the function lookup code to avoid calling memcmp() in these cases. I also added Asserts to catch any future violations of the new rule. Given the utter lack of any evidence that this actually causes any problems in the field, I don't feel a need to back-patch this change. Per report from Piotr Stefaniak, though this is not his patch. http://git.postgresql.org/pg/commitdiff/0a52d378b03b7d5ab1d64627a87edaf5ed311c6c

Peter Eisentraut pushed:

Heikki Linnakangas pushed:

  • Add missing newline to debug-message. Michael Paquier http://git.postgresql.org/pg/commitdiff/9cb36981fbbf2f298db2476101f4475c52d00fbb
  • Fix a couple of bugs with wal_log_hints. 1. Replay of the WAL record for setting a bit in the visibility map contained an assertion that a full-page image of that record type can only occur with checksums enabled. But it can also happen with wal_log_hints, so remove the assertion. Unlike checksums, wal_log_hints can be changed on the fly, so it would be complicated to figure out if it was enabled at the time that the WAL record was generated. 2. wal_log_hints has the same effect on the locking needed to read the LSN of a page as data checksums. BufferGetLSNAtomic() didn't get the memo. Backpatch to 9.4, where wal_log_hints was added. http://git.postgresql.org/pg/commitdiff/4b8e24b9ad308c30dbe2184e06848e638e018114
  • Fix typo in comment. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/7845db2aa778aa751b41cff72c41c94993e975e3
  • Add missing_ok option to the SQL functions for reading files. This makes it possible to use the functions without getting errors, if there is a chance that the file might be removed or renamed concurrently. pg_rewind needs to do just that, although this could be useful for other purposes too. (The changes to pg_rewind to use these functions will come in a separate commit.) The read_binary_file() function isn't very well-suited for extensions.c's purposes anymore, if it ever was. So bite the bullet and make a copy of it in extension.c, tailored for that use case. This seems better than the accidental code reuse, even if it's a some more lines of code. Michael Paquier, with plenty of kibitzing by me. http://git.postgresql.org/pg/commitdiff/cb2acb1081e13b4b27a76c6b5311115528e49c59
  • Don't choke on files that are removed while pg_rewind runs. If a file is removed from the source server, while pg_rewind is running, the invocation of pg_read_binary_file() will fail. Use the just-added missing_ok option to that function, to have it return NULL instead, and handle that gracefully. And similarly for pg_ls_dir and pg_stat_file. Reported by Fujii Masao, fix by Michael Paquier. http://git.postgresql.org/pg/commitdiff/b36805f3c54fe0e50e58bb9e6dad66daca46fbf6
  • Fix double-XLogBeginInsert call in GIN page splits. If data checksums or wal_log_hints is on, and a GIN page is split, the code to find a new, empty, block was called after having already called XLogBeginInsert(). That causes an assertion failure or PANIC, if finding the new block involves updating a FSM page that had not been modified since last checkpoint, because that update is WAL-logged, which calls XLogBeginInsert again. Nested XLogBeginInsert calls are not supported. To fix, rearrange GIN code so that XLogBeginInsert is called later, after finding the victim buffers. Reported by Jeff Janes. http://git.postgresql.org/pg/commitdiff/a45c70acf35e43257d86313dcbb7bb0e5201fab1
  • Promote the assertion that XLogBeginInsert() is not called twice into ERROR. Seems like cheap insurance for WAL bugs. A spurious call to XLogBeginInsert() in itself would be fairly harmless, but if there is any data registered and the insertion is not completed/cancelled properly, there is a risk that the data ends up in a wrong WAL record. Per Jeff Janes's suggestion. http://git.postgresql.org/pg/commitdiff/a32c3ec893cafbd3a4b42c34270a80198f28f123
  • Fix markup in docs. Oops. I could swear I built the docs before pushing, but I guess not.. http://git.postgresql.org/pg/commitdiff/6ab4d38ab085b0177d7ce63f7e1f2fb3f3a8e4a5

Fujii Masao pushed:

Andres Freund pushed:

  • Fix the fallback memory barrier implementation to be reentrant. This was essentially "broken" since 0c8eda62; but until more recently (14e8803f) barriers usage in signal handlers was infrequent. The failure to be reentrant was noticed because the test_shm_mq, which uses memory barriers at a high frequency, occasionally got stuck on some solaris buildfarm animals. Turns out, those machines use sun studio 12.1, which doesn't yet have efficient memory barrier support. A machine with a newer sun studio did not fail. Forcing the barrier fallback to be used on x86 allows to reproduce the problem. The new fallback is to use kill(PostmasterPid, 0) based on the theory that that'll always imply a barrier due to checking the liveliness of PostmasterPid on systems old enough to need fallback support. It's hard to come up with a good and performant fallback. I'm not backpatching this for now - the problem isn't active in the back branches, and we haven't backpatched barrier changes for now. Additionally master looks entirely different than the back branches due to the new atomics abstraction. It seems better to let this rest in master, where the non-reentrancy actively causes a problem, and then consider backpatching. Found-By: Robert Haas Discussion: 55626265.3060800@dunslane.net http://git.postgresql.org/pg/commitdiff/1b468a131bd260c9041484f78b8580c7f232d580
  • Fix test_decoding's handling of nonexistant columns in old tuple versions. test_decoding used fastgetattr() to extract column values. That's wrong when decoding updates and deletes if a table's replica identity is set to FULL and new columns have been added since the old version of the tuple was created. Due to the lack of a crosscheck with the datum's natts values an invalid value will be output, leading to errors or worse. Bug: #13470 Reported-By: Krzysztof Kotlarski Discussion: 20150626100333.3874.90852@wrigleys.postgresql.org Backpatch to 9.4, where the feature, including the bug, was added. http://git.postgresql.org/pg/commitdiff/d47a1136e441cebe7ae7fe72d70eb8ce278d5cd6

Ãlvaro Herrera pushed:

Simon Riggs pushed:

Kevin Grittner pushed:

  • Add opaque declaration of HTAB to tqual.h. Commit b89e151054a05f0f6d356ca52e3b725dd0505e53 added the ResolveCminCmaxDuringDecoding declaration to tqual.h, which uses an HTAB parameter, without declaring HTAB. It accidentally fails to fail to build with current sources because a declaration happens to be included, directly or indirectly, in all source files that currently use tqual.h before tqual.h is first included, but we shouldn't count on that. Since an opaque declaration is enough here, just use that, as was done in snapmgr.h. Backpatch to 9.4, where the HTAB reference was added to tqual.h. http://git.postgresql.org/pg/commitdiff/604e99396de02f6f23950ee373c13335d2ccdf05
  • Fix comment for GetCurrentIntegerTimestamp(). The unit of measure is microseconds, not milliseconds. Backpatch to 9.3 where the function and its comment were added. http://git.postgresql.org/pg/commitdiff/cca8ba9529f8815acd23fe88c32763765d0e1b68

Tatsuo Ishii pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Peter Geoghegan sent in a patch to allow JSON[B] arrays to take negative subscripts.

Tomas Vondra sent in a patch to add density correction to sampling for statistics collection.

Michael Paquier sent in a patch to fix some white space in pg_rewind error messages.

Michael Paquier and Robert Haas traded patches to fix an issue where a dangerous rm -rf was being issued in the global makefile.

Michael Paquier sent in two more revisions of a patch to update the hash index creation warning.

Abhijit Menon-Sen sent in a patch to introduce XLogLockBlockRangeForCleanup().

Fabien COELHO sent in another revision of a patch to add checkpointer continuous flushing.

Michael Paquier sent in another revision of a patch to add support for TAP tests on Windows.

Michael Paquier sent in a patch to improve log capture of TAP tests and fix race conditions.

Fabrízio de Royes Mello sent in two more revisions of a patch to add CINE for ALTER TABLE ... ADD COLUMN.

Uriy Zhuravlev sent in another revision of a patch to implement ALTER OPERATOR.

Craig Ringer sent in a patch to implement ALTER TABLE ... ALTER CONSTRAINT ... SET DEFERRABLE on UNIQUE or PK.

Oskari Saarenmaa sent in a patch to add -lrt to configure for sched_yield on Solaris.

Jim Nasby sent in a patch to ensure that object_classes is properly sized.

Marco Nenciarini sent in a patch to fix an off-by-one bug which caused VACUUM FREEZE to mistakenly cancel standby sessions.

Peter Geoghegan sent in a patch to add some compatibility notes for UPSERT with foreign data wrappers.

Amit Kapila sent in a patch to improve the performance of DROP TABLE when the shared_buffers setting is high.

Jeff Janes sent in a patch to make pg_trgm perform better by supporting the triconsistent function, introduced in version 9.4 of the server, to make it faster to implement indexed queries where some keys are common and some are rare.

Tom Lane sent in a patch to refactor the way the pg_file_settings view works.

par N Bougain le lundi 29 juin 2015 à 21h21

samedi 27 juin 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 21 juin 2015

La PostgreSQL Conference Europe 2015 qui aura lieu du 27 au 30 octobre accepte à présent les inscriptions des auditeurs : http://2015.pgconf.eu/registration/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juin

PostgreSQL Local

  • La deuxième Conférence PostgreSQL Suisse se tiendra à Rapperswil les 25 & 26 juin 2015 : http://www.postgres-conference.ch/
  • La conférence PGDay UK aura lieu le 7 juillet 2015 - elle vise les membres de la communauté PostgreSQL anglaise. L'appel à conférenciers expire le 13 avril : http://www.postgresqlusergroup.org.uk
  • Le PGDay Campinas 2015 aura lieu à Campinas (Brésil) le 7 août. L'appel à conférenciers expire le 31 mai : http://pgdaycampinas.com.br/english/
  • L'appel à conférenciers pour le PostgresOpen 2015, programmé à Dallas (Texas) du 16 au 18 septembre, a été lancé : http://2015.postgresopen.org/callforpapers/
  • L'appel à conférenciers pour la session PostgreSQL n°7 (24 septembre 2015 à Paris) est lancé jusqu'au 15 juin 2015 : call-for-paper <AT> postgresql-sessions <DOT> org : http://www.postgresql-sessions.org/7/about
  • PostgreSQL Conference Europe 2015 aura lieu du 27 au 30 octobre au Vienna Marriott Hotel à Vienne (Autriche). L'appel à conférenciers est lancé jusqu'au 7 août : http://2015.pgconf.eu/
  • PGConf Silicon Valley 2015 se tiendra au centre de convention sud de San Francisco les 17 & 18 novembre. L'appel à conférenciers porte jusqu'au 15 juin : http://www.pgconfsv.com

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150621221026.GB32298@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Michael Meskes pushed:

Andrew Dunstan pushed:

Noah Misch pushed:

  • Detect setlocale(LC_CTYPE, NULL) clobbering previous return values. POSIX permits setlocale() calls to invalidate any previous setlocale() return values. Commit 5f538ad004aa00cf0881f179f0cde789aad4f47e neglected to account for that. In advance of fixing that bug, switch to failing hard on affected configurations. This is a planned temporary commit to assay buildfarm-represented configurations. http://git.postgresql.org/pg/commitdiff/b76e76be460a240e99c33f6fb470dd1d5fe01a2a
  • Revert "Detect setlocale(LC_CTYPE, NULL) clobbering previous return values." This reverts commit b76e76be460a240e99c33f6fb470dd1d5fe01a2a. The buildfarm yielded no related failures. http://git.postgresql.org/pg/commitdiff/1f2a378de41bf3e516b8d2c4d65790aeefbfb89d
  • Fix failure to copy setlocale() return value. POSIX permits setlocale() calls to invalidate any previous setlocale() return values, but commit 5f538ad004aa00cf0881f179f0cde789aad4f47e neglected to account for setlocale(LC_CTYPE, NULL) doing so. The effect was to set the LC_CTYPE environment variable to an unintended value. pg_perm_setlocale() sets this variable to assist PL/Perl; without it, Perl would undo PostgreSQL's locale settings. The known-affected configurations are 32-bit, release builds using Visual Studio 2012 or Visual Studio 2013. Visual Studio 2010 is unaffected, as were all buildfarm-attested configurations. In principle, this bug could leave the wrong LC_CTYPE in effect after PL/Perl use, which could in turn facilitate problems like corrupt tsvector datums. No known platform experiences that consequence, because PL/Perl on Windows does not use this environment variable. The bug has been user-visible, as early postmaster failure, on systems with Windows ANSI code page set to CP936 for "Chinese (Simplified, PRC)" and probably on systems using other multibyte code pages. (SetEnvironmentVariable() rejects values containing character data not valid under the Windows ANSI code page.) Back-patch to 9.4, where the faulty commit first appeared. Reported by Didi Hu and æž—é¹ç¨‹. Reviewed by Tom Lane, though this fix strategy was not his first choice. http://git.postgresql.org/pg/commitdiff/f0a264a362343287051c4737b01aa3ebe36f21a6

Robert Haas pushed:

Tom Lane pushed:

  • Fix bogus range_table_mutator() logic for RangeTblEntry.tablesample. Must make a copy of the TableSampleClause node; the previous coding modified the input data structure in-place. Petr Jelinek http://git.postgresql.org/pg/commitdiff/be87143fe90adf8862791aeddd76151e88ce5603
  • In immediate shutdown, postmaster should not exit till children are gone. This adjusts commit 82233ce7ea42d6ba519aaec63008aff49da6c7af so that the postmaster does not exit until all its child processes have exited, even if the 5-second timeout elapses and we have to send SIGKILL. There is no great value in having the postmaster process quit sooner, and doing so can mislead onlookers into thinking that the cluster is fully terminated when actually some child processes still survive. This effect might explain recent test failures on buildfarm member hamster, wherein we failed to restart a cluster just after shutting it down with "pg_ctl stop -m immediate". I also did a bit of code review/beautification, including fixing a faulty use of the Max() macro on a volatile expression. Back-patch to 9.4. In older branches, the postmaster never waited for children to exit during immediate shutdowns, and changing that would be too much of a behavioral change. http://git.postgresql.org/pg/commitdiff/48913db887e6a41fa3f1b6cdf80ee89e38f21d9d

Ãlvaro Herrera pushed:

Peter Eisentraut pushed:

Andres Freund pushed:

  • Add missing check for wal_debug GUC. 9a20a9b2 added a new elog(), enabled when WAL_DEBUG is defined. The other WAL_DEBUG dependant messages check for the wal_debug GUC, but this one did not. While at it replace 'upto' with 'up to'. Discussion: 20150610110253.GF3832@alap3.anarazel.de Backpatch to 9.4, the first release containing 9a20a9b2. http://git.postgresql.org/pg/commitdiff/90231cd5188c43da94f58f7a839eee9286d0f864
  • Improve multixact emergency autovacuum logic. Previously autovacuum was not necessarily triggered if space in the members slru got tight. The first problem was that the signalling was tied to values in the offsets slru, but members can advance much faster. Thats especially a problem if old sessions had been around that previously prevented the multixact horizon to increase. Secondly the skipping logic doesn't work if the database was restarted after autovacuum was triggered - that knowledge is not preserved across restart. This is especially a problem because it's a common panic-reaction to restart the database if it gets slow to anti-wraparound vacuums. Fix the first problem by separating the logic for members from offsets. Trigger autovacuum whenever a multixact crosses a segment boundary, as the current member offset increases in irregular values, so we can't use a simple modulo logic as for offsets. Add a stopgap for the second problem, by signalling autovacuum whenver ERRORing out because of boundaries. Discussion: 20150608163707.GD20772@alap3.anarazel.de Backpatch into 9.3, where it became more likely that multixacts wrap around. http://git.postgresql.org/pg/commitdiff/667912aee649c3608e003568e4b47d95251b1c8c

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

David Rowley sent in a patch to let the executor notice when multiple aggregates in a query share a transition function, executing the common ones only once per row.

David Rowley sent in another revision of a patch to improve some appendStringInfo* calls.

Petr Jelinek sent in a patch to extend CREATE EXTENSION to include its dependencies via the optional RECURSIVE keyword.

Fabien COELHO sent in another revision of a patch to fix pgbench --progress report under (very) low rate.

Tomas Vondra sent in a patch to teach the expression walker about RestrictInfo.

Amit Kapila sent in a patch to rename mapfile if backupfile not present.

Vik Fearing sent in a patch to add tab completion to psql for CREATE SEQUENCE.

Michael Paquier sent in a patch to reproduce a problem with pg_rewind.

Michael Paquier sent in a patch to make process_remote_file ignore files named as pg_xlog/xlogtemp.*.

SAWADA Masahiko sent in a patch to give the GIN function of pageinspect a consistent data type.

Fabien COELHO sent in another revision of a patch to smooth checkpoint flushing.

Tomas Vondra sent in another revision of a patch to make a better ndistinct estimator.

Brendan Jurd sent in two revisions of a patch to add a new built-in function pg_notify_queue_saturation().

Michael Paquier sent in three more revisions of a patch to improve the way TAP tests log their output using IPC::Run::run.

Michael Paquier sent in a patch to add a missing -w switch to the pg_ctl stop call in pg_regress.

Robert Haas, Tom Lane, and Dean Rasheed traded patches to fix some infelicities recently introduced in the table inheritance part of the planner.

Marti Raudsepp sent in a patch to fix pg_upgrade when postgres template1 aren't in the default database.

Petr Jelinek sent in another revision of a patch to implement tab completion in psql for TABLESAMPLE.

Ãlvaro Herrera sent in a patch to add pg_get_multixact_members() and pg_get_multixact_range().

Thomas Munro sent in a patch to fix an issue where the get_relation_info comment was out of sync with the code nearby.

Andres Freund sent in a patch to rework the way multixact truncations work with the goal of making them WAL logged.

Fabien COELHO sent in a patch to allow backslash-continuations in custom scripts.

par N Bougain le samedi 27 juin 2015 à 00h40

Nouvelles hebdomadaires de PostgreSQL - 14 juin 2015

Mises à jour correctives 9.4.4, 9.3.9, 9.2.13, 9.1.18 et 9.0.22 publiées. Déployez ! http://www.postgresql.org/docs/current/static/release.html

La classe d'administration de PostgreSQL de Consistent State aura lieu à Dallas (Texas, USA) du 21 au 24 septembre 2015 : http://www.consistentstate.com/store/p8/PostgreSQL_Admin_Class%3B_Dallas.html

Gianni Ciolli fera l'une des présentations introductives à la Dynamic Languages Conference du 20 juin à Manchester (Royaume-Uni) : http://www.dynamiclanguages.co.uk/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juin

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150614224208.GA22327@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Peter Eisentraut pushed:

Andrew Dunstan pushed:

  • Desupport jsonb subscript deletion on objects. Supporting deletion of JSON pairs within jsonb objects using an array-style integer subscript allowed for surprising outcomes. This was mostly due to the implementation-defined ordering of pairs within objects for jsonb. It also seems desirable to make jsonb integer subscript deletion consistent with the 9.4 era general purpose integer subscripting operator for jsonb (although that operator returns NULL when an object is encountered, while we prefer here to throw an error). Peter Geoghegan, following discussion on -hackers. http://git.postgresql.org/pg/commitdiff/b81c7b4098f52e64df89efe1461ba00a54649a10
  • Clarify documentation of jsonb - text. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/94d6727dbe61117addd9c24eea28440a2151ccf4
  • Rename jsonb - text[] operator to #- to avoid ambiguity. Following recent discussion on -hackers. The underlying function is also renamed to jsonb_delete_path. The regression tests now don't need ugly type casts to avoid the ambiguity, so they are also removed. Catalog version bumped. http://git.postgresql.org/pg/commitdiff/908e234733574545866045c7d5f93d4dd50ef66d
  • Fix "path" infrastructure bug affecting jsonb_set(). jsonb_set() and other clients of the setPathArray() utility function could get spurious results when an array integer subscript is provided that is not within the range of int. To fix, ensure that the value returned by strtol() within setPathArray() is within the range of int; when it isn't, assume an invalid input in line with existing, similar cases. The path-orientated operators that appeared in PostgreSQL 9.3 and 9.4 do not call setPathArray(), and already independently take this precaution, so no change there. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/2271d002d5c305441398e8f7a295f18ec3c132a9

Andres Freund pushed:

  • Allow HotStandbyActiveInReplay() to be called in single user mode. HotStandbyActiveInReplay, introduced in 061b079f, only allowed WAL replay to happen in the startup process, missing the single user case. This buglet is fairly harmless as it only causes problems when single user mode in an assertion enabled build is used to replay a btree vacuum record. Backpatch to 9.2. 061b079f was backpatched further, but the assertion was not. http://git.postgresql.org/pg/commitdiff/d1b958218ac183d0e88348341ff6ba31397086ad

Fujii Masao pushed:

  • Refactor WAL segment copying code. * Remove unused argument "dstfname" and related code from XLogFileCopy(). * Previously XLogFileCopy() returned a pstrdup'd string so that InstallXLogFileSegment() used it later. Since the pstrdup'd string was never free'd, there could be a risk of memory leak. It was almost harmless because the startup process exited just after calling XLogFileCopy(), it existed. This commit changes XLogFileCopy() so that it directly calls InstallXLogFileSegment() and doesn't call pstrdup() at all. Which fixes that memory leak problem. * Extend InstallXLogFileSegment() so that the caller can specify the log level. Which allows us to emit an error when InstallXLogFileSegment() fails a disk file access like link() and rename(). Previously it was always logged with LOG level and additionally needed to be logged with ERROR when we wanted to treat it as an error. Michael Paquier http://git.postgresql.org/pg/commitdiff/7abc68597436da1475b4d9b08f4fa9f3c5ed6185
  • Fix typo in comment. David Rowley http://git.postgresql.org/pg/commitdiff/ea9c4c1e4a7a9b602d867dcb02e07ef1fe51f6ec
  • Fix some issues in pg_rewind. * Remove invalid option character "N" from the third argument (valid option string) of getopt_long(). * Use pg_free() or pfree() to free the memory allocated by pg_malloc() or palloc() instead of always using free(). * Assume problem is no disk space if write() fails but doesn't set errno. * Fix several typos. Patch by me. Review by Michael Paquier. http://git.postgresql.org/pg/commitdiff/966c37fdb5ed9b87f3e91eace4dbbed7909f6769
  • Clean up useless mention of RMGRDESCSOURCES in pg_rewind Makefile. RMGRDESCSOURCES is defined and used only in pg_xlogdump Makefile, but pg_rewind Makefile mentioned it as extra files to remove in "make clean". This patch removes that useless mention from pg_rewind Makefile. Michael Paquier http://git.postgresql.org/pg/commitdiff/cd3cff4778e011c584e1481a6803dec5d4756a6e
  • Fix alphabetization in catalogs.sgml. System catalogs and views should be listed alphabetically in catalog.sgml, but only pg_file_settings view not. This patch also fixes typos in pg_file_settings comments. http://git.postgresql.org/pg/commitdiff/091c02a958fd0ae02b96492d9728efe8526385e6
  • Make postmaster restart archiver soon after it dies, even during recovery. After the archiver dies, postmaster tries to start a new one immediately. But previously this could happen only while server was running normally even though archiving was enabled always (i.e., archive_mode was set to always). So the archiver running during recovery could not restart soon after it died. This is an oversight in commit ffd3774. This commit changes reaper(), postmaster's signal handler to cleanup after a child process dies, so that it tries to a new archiver even during recovery if necessary. Patch by me. Review by Alvaro Herrera. http://git.postgresql.org/pg/commitdiff/b5fe62038f49f92c4a4f189c7cdacf3739effcdc

Ãlvaro Herrera pushed:

Tom Lane pushed:

  • First-draft release notes for 9.4.4, 9.3.9, 9.2.13, 9.1.18, 9.0.22. http://git.postgresql.org/pg/commitdiff/21187cfc7dfd82461db9119377a76366c00d27c3
  • Report more information if pg_perm_setlocale() fails at startup. We don't know why a few Windows users have seen this fail, but the taciturnity of the error message certainly isn't helping debug it. Let's at least find out which LC category isn't working. http://git.postgresql.org/pg/commitdiff/f6e9cbfd9153958226b27e31d658e7b64351c71f
  • Release notes for 9.4.4, 9.3.9, 9.2.13, 9.1.18, 9.0.22. http://git.postgresql.org/pg/commitdiff/b94085920b016e64ee40a0f6c50199889565cc56
  • Improve error message and hint for ALTER COLUMN TYPE can't-cast failure. We already tried to improve this once, but the "improved" text was rather off-target if you had provided a USING clause. Also, it seems helpful to provide the exact text of a suggested USING clause, so users can just copy-and-paste it when needed. Per complaint from Keith Rarick and a suggestion from Merlin Moncure. Back-patch to 9.2 where the current wording was adopted. http://git.postgresql.org/pg/commitdiff/b00982344a73d9cb626430dd17a6da84c15c9980
  • Fix failure to cover scalar-vs-rowtype cases in exec_stmt_return(). In commit 9e3ad1aac52454569393a947c06be0d301749362 I modified plpgsql to use exec_stmt_return's simple-variables fast path in more cases. However, I overlooked that there are really two different return conventions in use here, depending on whether estate->retistuple is true, and the existing fast-path code had only bothered to handle one of them. So trying to return a scalar in a function returning composite, or vice versa, could lead to unexpected error messages (typically "cache lookup failed for type 0") or to a null-pointer-dereference crash. In the DTYPE_VAR case, we can just throw error if retistuple is true, corresponding to what happens in the general-expression code path that was being used previously. (Perhaps someday both of these code paths should attempt a coercion, but today is not that day.) In the REC and ROW cases, just hand the problem to exec_eval_datum() when not retistuple. Also clean up the ROW coding slightly so it looks more like exec_eval_datum(). The previous commit also caused exec_stmt_return_next() to be used in more cases, but that code seems to be OK as-is. Per off-list report from Serge Rielau. This bug is new in 9.5 so no need to back-patch. http://git.postgresql.org/pg/commitdiff/ae58f1430abb4b0c309c40b377f91bf9d080334b

Bruce Momjian pushed:

Kevin Grittner pushed:

Michael Meskes pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Michael Paquier sent in two more revisions of a patch to fix some memory leaks in pg_dump, initdb and pg_upgrade.

Abhijit Menon-Sen sent in another revision of a patch to skip files in pg_log during basebackup.

Thomas Munro sent in two revisions of a patch to fix bogus truncation errors.

Dean Rasheed sent in a patch to fix an incompatibility between RLS and WHERE CURRENT OF.

Michael Paquier sent in a patch to restrict visibility of SSL information in pg_stat_ssl.

Andres Freund sent in a patch to improve multixact emergency autovacuum logic.

Jeevan Chalke sent in a patch to remove dead code in Create/RenameRole() after RoleSpec changes related to CURRENT/SESSION_USER.

Jeevan Chalke sent in another revision of a patch to add \ev and \sv (edit- and show-view) functionality to psql.

Michael Paquier sent in a patch to expand the use of xlog_internal.h's macros.

Alexander Shulgin sent in a patch to fix a problem where logical decoding sendtime update was always set to zero.

Gurjeet Singh sent in three revisions of a patch to initialize restart_lsn when the replication slot is created.

Peter Geoghegan sent in a patch to refactor speculative insertion with unique indexes.

Amit Kapila sent in another revision of a patch to remove only symlinks during recovery.

Etsuro Fujita sent in a patch to fix some comments in table inheritance.

Etsuro Fujita sent in a patch to fix a typo in comment in setrefs.c.

Jan Wieck sent in a patch to change some of the s_lock parameters with the aim of improving performance on modern NUMA machines.

Kaigai Kouhei sent in another revision of a patch to implement custom-join children.

Kaigai Kouhei sent in a patch to fix an issue where DBT-3 with SF=20 failed.

Naoya Anzai sent in a patch to make finding the backend PID a shortcut, \bid.

Andrew Dunstan sent in a patch to add json_count_array().

Peter Geoghegan sent in a patch to remove dead code for jsonb subscript deletion.

Peter Geoghegan sent in a patch to fully remove heap_formtuple() and friends.

Pavel Stehule sent in a patch to lower the log level for success in (un)registering background workers from LOG to DEBUG1.

Petr Jelínek sent in a patch to add tab completion for TABLESAMPLE to psql.

Dean Rasheed sent in a patch to add documentation for CREATE FUNCTION .. LEAKPROOF.

Ãlvaro Herrera sent in a patch to add a pg_get_multixact_range() function.

Petr Jelinek send in a flock of patches which: splits out pg_resetxlog into some reusable pieces and the calls to same, adds pg_resetsysid utility which changes the system id to newly generated one, adds -s option to pg_resetxlog to change the system id to the one specified, and adds fall-back implementation of the two functions based on very slightly modified code from OpenBSD.

Jeff Davis sent in another revision of a patch to implement memory accounting by tracking memory usage (at the block level) for all memory contexts.

par N Bougain le samedi 27 juin 2015 à 00h31

Nouvelles hebdomadaires de PostgreSQL - 7 juin 2015

Mises à jour correctives 9.4.3, 9.3.8, 9.2.12, 9.1.17 et 9.0.21 publiées. Déployez ! https://wiki.postgresql.org/wiki/May_2015_Fsync_Permissions_Bug

PGBR2015 (la PgConf brésilienne) aura lieu à Porto Alegre (État du Rio Grande do Sul) les 18, 19 et 20 novembre. L'appel à conférenciers expire le 15 juillet : http://pgbr.postgresql.org.br/2015/en/#call-for-papers

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en juin

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150608005708.GA17798@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Andrew Dunstan pushed:

Bruce Momjian pushed:

Tom Lane pushed:

  • Release notes for 9.4.3, 9.3.8, 9.2.12, 9.1.17, 9.0.21. Also sneak entries for commits 97ff2a564 et al into the sections for the previous releases in the relevant branches. Those fixes did go out in the previous releases, but missed getting documented. http://git.postgresql.org/pg/commitdiff/82ec7d28211b97a2c9917b7a71edbe6b019578da
  • Fix planner's cost estimation for SEMI/ANTI joins with inner indexscans. When the inner side of a nestloop SEMI or ANTI join is an indexscan that uses all the join clauses as indexquals, it can be presumed that both matched and unmatched outer rows will be processed very quickly: for matched rows, we'll stop after fetching one row from the indexscan, while for unmatched rows we'll have an indexscan that finds no matching index entries, which should also be quick. The planner already knew about this, but it was nonetheless charging for at least one full run of the inner indexscan, as a consequence of concerns about the behavior of materialized inner scans --- but those concerns don't apply in the fast case. If the inner side has low cardinality (many matching rows) this could make an indexscan plan look far more expensive than it actually is. To fix, rearrange the work in initial_cost_nestloop/final_cost_nestloop so that we don't add the inner scan cost until we've inspected the indexquals, and then we can add either the full-run cost or just the first tuple's cost as appropriate. Experimentation with this fix uncovered another problem: add_path and friends were coded to disregard cheap startup cost when considering parameterized paths. That's usually okay (and desirable, because it thins the path herd faster); but in this fast case for SEMI/ANTI joins, it could result in throwing away the desired plain indexscan path in favor of a bitmap scan path before we ever get to the join costing logic. In the many-matching-rows cases of interest here, a bitmap scan will do a lot more work than required, so this is a problem. To fix, add a per-relation flag consider_param_startup that works like the existing consider_startup flag, but applies to parameterized paths, and set it for relations that are the inside of a SEMI or ANTI join. To make this patch reasonably safe to back-patch, care has been taken to avoid changing the planner's behavior except in the very narrow case of SEMI/ANTI joins with inner indexscans. There are places in compare_path_costs_fuzzily and add_path_precheck that are not terribly consistent with the new approach, but changing them will affect planner decisions at the margins in other cases, so we'll leave that for a HEAD-only fix. Back-patch to 9.3; before that, the consider_startup flag didn't exist, meaning that the second aspect of the patch would be too invasive. Per a complaint from Peter Holzer and analysis by Tomas Vondra. http://git.postgresql.org/pg/commitdiff/3f59be836c555fa679bbe0ec76de50a8b5cb23e0
  • Fix some questionable edge-case behaviors in add_path() and friends. add_path_precheck was doing exact comparisons of path costs, but it really needs to do them fuzzily to be sure it won't reject paths that could survive add_path's comparisons. (This can only matter if the initial cost estimate is very close to the final one, but that turns out to often be true.) Also, it should ignore startup cost for this purpose if and only if compare_path_costs_fuzzily would do so. The previous coding always ignored startup cost for parameterized paths, which is wrong as of commit 3f59be836c555fa6; it could result in improper early rejection of paths that we care about for SEMI/ANTI joins. It also always considered startup cost for unparameterized paths, which is just as wrong though the only effect is to waste planner cycles on paths that can't survive. Instead, it should consider startup cost only when directed to by the consider_startup/ consider_param_startup relation flags. Likewise, compare_path_costs_fuzzily should have symmetrical behavior for parameterized and unparameterized paths. In this case, the best answer seems to be that after establishing that total costs are fuzzily equal, we should compare startup costs whether or not the consider_xxx flags are on. That is what it's always done for unparameterized paths, so let's make the behavior for parameterized paths match. These issues were noted while developing the SEMI/ANTI join costing fix of commit 3f59be836c555fa6, but we chose not to back-patch these fixes, because they can cause changes in the planner's choices among nearly-same-cost plans. (There is in fact one minor change in plan choice within the core regression tests.) Destabilizing plan choices in back branches without very clear improvements is frowned on, so we'll just fix this in HEAD. http://git.postgresql.org/pg/commitdiff/3b0f77601b9f9f3a2e36a813e4cd32c00e0864d6
  • Stabilize query plans in rowsecurity regression test. Some recent buildfarm failures can be explained by supposing that autovacuum or autoanalyze fired on the tables created by this test, resulting in plan changes. Do a proactive VACUUM ANALYZE on the test's principal tables to try to forestall such changes. http://git.postgresql.org/pg/commitdiff/5cdf25e16843dff33dbc2ddc02941458032e3ad4
  • Stabilize results of brin regression test. This test used seqscans on tenk1, with LIMIT, to build test data. That works most of the time, but if the synchronized-seqscan logic kicks in, we get varying test data. This seems likely to explain the erratic test failures on buildfarm member chipmunk, which uses smaller-than-default shared_buffers. To fix, add ORDER BY clauses to force the ordering to be what it was implicitly being assumed to be. Peter Geoghegan had noticed this with respect to one of the trouble spots, though not the ones actually causing the chipmunk issue. http://git.postgresql.org/pg/commitdiff/bac99475eb6e9e6d69a91fee30b5420b8e0115be
  • Fix brin "char" test to actually test what it meant to test. Casting to char, without quotes, does not give the same results as casting to "char". That meant we were not testing the brin "char" paths at all, since we ended up with a text operator not a "char" operator. http://git.postgresql.org/pg/commitdiff/78e72794a76fef3233c06800c6046aaad0704a22
  • Tighten the per-operator testing done in brin regression test. Verify that the number of matches is exactly what it should be, not just that it not be zero. This should help us detect any environment-dependent issues. Also, verify that we're getting the expected type of scan plan (either bitmap or seqscan as appropriate). Right now, this is failing on the cidrcol test cases, as shown in the output file. I'll look into that in a bit, but it seems good to commit this as-is temporarily to verify that it behaves as expected on the buildfarm. http://git.postgresql.org/pg/commitdiff/79454c696bd3346a9f00f5e940398fb01a337fad
  • Fix brin regression test so it actually tests cidr. The problem noted in my previous commit was simpler than I thought: we weren't getting an index plan because the column wasn't indexed. http://git.postgresql.org/pg/commitdiff/1676e4381f48f7bf211f0965ad23abe10a683818
  • Second try at stabilizing query plans in rowsecurity regression test. This reverts commit 5cdf25e16843dff33dbc2ddc02941458032e3ad4, which was almost immediately proven insufficient by the buildfarm. On second thought, the tables involved are not large enough that autovacuum or autoanalyze would notice them; what seems far more likely to be the culprit is the database-wide "vacuum analyze" in the concurrent gist test. That thing has given us one headache too many, so get rid of it in favor of targeted vacuuming of that test's own tables only. http://git.postgresql.org/pg/commitdiff/1d27842519999cbac7e1cca8beaef053be9c7825
  • Fix incorrect order of database-locking operations in InitPostgres(). We should set MyProc->databaseId after acquiring the per-database lock, not beforehand. The old way risked deadlock against processes trying to copy or delete the target database, since they would first acquire the lock and then wait for processes with matching databaseId to exit; that left a window wherein an incoming process could set its databaseId and then block on the lock, while the other process had the lock and waited in vain for the incoming process to exit. CountOtherDBBackends() would time out and fail after 5 seconds, so this just resulted in an unexpected failure not a permanent lockup, but it's still annoying when it happens. A real-world example of a use-case is that short-duration connections to a template database should not cause CREATE DATABASE to fail. Doing it in the other order should be fine since the contract has always been that processes searching the ProcArray for a database ID must hold the relevant per-database lock while searching. Thus, this actually removes the former race condition that required an assumption that storing to MyProc->databaseId is atomic. It's been like this for a long time, so back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/ac23b711dd6ccb82fb70ca0f153fe755fd809a46
  • Get rid of a //-style comment. Not sure how "//XXX" got into a committed patch in the first place, as it's both content-free and against project style. pgindent made a bit of a hash of it, too. Going forward, we should have at least one buildfarm member using "gcc -ansi" to catch such things, at least till such time as we decide the project target language isn't C90 any more. I've turned this option on on dromedary. http://git.postgresql.org/pg/commitdiff/1497369e5df8bb129256f677a85327f80d3767d3
  • Use a safer method for determining whether relcache init file is stale. When we invalidate the relcache entry for a system catalog or index, we must also delete the relcache "init file" if the init file contains a copy of that rel's entry. The old way of doing this relied on a specially maintained list of the OIDs of relations present in the init file: we made the list either when reading the file in, or when writing the file out. The problem is that when writing the file out, we included only rels present in our local relcache, which might have already suffered some deletions due to relcache inval events. In such cases we correctly decided not to overwrite the real init file with incomplete data --- but we still used the incomplete initFileRelationIds list for the rest of the current session. This could result in wrong decisions about whether the session's own actions require deletion of the init file, potentially allowing an init file created by some other concurrent session to be left around even though it's been made stale. Since we don't support changing the schema of a system catalog at runtime, the only likely scenario in which this would cause a problem in the field involves a "vacuum full" on a catalog concurrently with other activity, and even then it's far from easy to provoke. Remarkably, this has been broken since 2002 (in commit 786340441706ac1957a031f11ad1c2e5b6e18314), but we had never seen a reproducible test case until recently. If it did happen in the field, the symptoms would probably involve unexpected "cache lookup failed" errors to begin with, then "could not open file" failures after the next checkpoint, as all accesses to the affected catalog stopped working. Recovery would require manually removing the stale "pg_internal.init" file. To fix, get rid of the initFileRelationIds list, and instead consult syscache.c's list of relations used in catalog caches to decide whether a relation is included in the init file. This should be a tad more efficient anyway, since we're replacing linear search of a list with ~100 entries with a binary search. It's a bit ugly that the init file contents are now so directly tied to the catalog caches, but in practice that won't make much difference. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/f3b5565dd4e59576be4c772da364704863e6a835

Fujii Masao pushed:

Peter Eisentraut pushed:

Robert Haas pushed:

  • docs: Fix list of object types pg_table_is_visible() can handle. Materialized views and foreign tables were missing from the list, probably because they are newer than the other object types that were mentioned. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/1c645da8ebb5532105481ad77bb1d9a671b1f086
  • doc: Session identifiers truncate, not round, the backend start time. Joel Jacobson http://git.postgresql.org/pg/commitdiff/99cfd5e136e2a20c77021390a1378d18a24b7388
  • Cope with possible failure of the oldest MultiXact to exist. Recent commits, mainly b69bf30b9bfacafc733a9ba77c9587cf54d06c0c and 53bb309d2d5a9432d2602c93ed18e58bd2924e15, introduced mechanisms to protect against wraparound of the MultiXact member space: the number of multixacts that can exist at one time is limited to 2^32, but the total number of members in those multixacts is also limited to 2^32, and older code did not take care to enforce the second limit, potentially allowing old data to be overwritten while it was still needed. Unfortunately, these new mechanisms failed to account for the fact that the code paths in which they run might be executed during recovery or while the cluster was in an inconsistent state. Also, they failed to account for the fact that users who used pg_upgrade to upgrade a PostgreSQL version between 9.3.0 and 9.3.4 might have might oldestMultiXid = 1 in the control file despite the true value being larger. To fix these problems, first, avoid unnecessarily examining the mmembers of MultiXacts when the cluster is not known to be consistent. TruncateMultiXact has done this for a long time, and this patch does not fix that. But the new calls used to prevent member wraparound are not needed until we reach normal running, so avoid calling them earlier. (SetMultiXactIdLimit is actually called before InRecovery is set, so we can't rely on that; we invent our own multixact-specific flag instead.) Second, make failure to look up the members of a MultiXact a non-fatal error. Instead, if we're unable to determine the member offset at which wraparound would occur, postpone arming the member wraparound defenses until we are able to do so. If we're unable to determine the member offset that should force autovacuum, force it continuously until we are able to do so. If we're unable to deterine the member offset at which we should truncate the members SLRU, log a message and skip truncation. An important consequence of these changes is that anyone who does have a bogus oldestMultiXid = 1 value in pg_control will experience immediate emergency autovacuuming when upgrading to a release that contains this fix. The release notes should highlight this fact. If a user has no pg_multixact/offsets/0000 file, but has oldestMultiXid = 1 in the control file, they may wish to vacuum any tables with relminmxid = 1 prior to upgrading in order to avoid an immediate emergency autovacuum after the upgrade. This must be done with a PostgreSQL version 9.3.5 or newer and with vacuum_multixact_freeze_min_age and vacuum_multixact_freeze_table_age set to 0. This patch also adds an additional log message at each database server startup, indicating either that protections against member wraparound have been engaged, or that they have not. In the latter case, once autovacuum has advanced oldestMultiXid to a sane value, the message indicating that the guards have been engaged will appear at the next checkpoint. A few additional messages have also been added at the DEBUG1 level so that the correct operation of this code can be properly audited. Along the way, this patch fixes another, related bug in TruncateMultiXact that has existed since PostgreSQL 9.3.0: when no MultiXacts exist at all, the truncation code looks up NextMultiXactId, which doesn't exist yet. This can lead to TruncateMultiXact removing every file in pg_multixact/offsets instead of keeping one around, as it should. This in turn will cause the database server to refuse to start afterwards. Patch by me. Review by Ãlvaro Herrera, Andres Freund, Noah Misch, and Thomas Munro. http://git.postgresql.org/pg/commitdiff/068cfadf9e2190bdd50a30d19efc7c9f0b825b5e

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Peter Geoghegan sent in a patch to illustrate a possible security bug in RLS.

Dean Rasheed sent in a patch to refactor the implementation of RLS.

Laurenz Albe sent in a patch to allow using literal tabs in psql.

Etsuro Fujita sent in a doc patch to add materialized views and foreign tables to database objects that pg_table_is_visible() can be used with.

Michael Paquier sent in two more revisions of a patch to fix a memory leak in XLogFileCopy.

Fabien COELHO sent in two revisions of a patch to enable flushing while writing during checkpoints.

Andreas Seltenreich sent in a patch to add error handling to byteaout.

Michael Paquier sent in a patch to remove the use of %.*s in several parts of the psql code to make the code more solid when facing non-ASCII strings.

Craig Ringer sent in another revision of a patch to allow sampling of only some queries by auto_explain.

Peter Geoghegan sent in a patch to desupport jsonb subscript deletion on objects, which was causing surprising outcomes.

Jeevan Chalke sent in another revision of a patch to implement a two-argument version of current_setting() with fallback.

Kaigai Kouhei sent in another revision of a patch to allow custom-join children.

Amit Kapila and Andrew Dunstan traded patches to remove only symlinks during recovery.

Julien Rouhaud sent in a patch fix an issue where when archiver aborts, pg_stat_archiver doesn't report those failed attempts.

Petr Korobeinikov sent in another revision of a patch to implement \ev and \sv (edit view, and show view, respectively) in psql.

Peter Geoghegan sent in a patch to add a regression test in cases where RLS again fails to play nicely with UPDATE ... WHERE CURRENT OF.

Thomas Munro sent in a patch to fix a bogus subtrans wraparound error.

par N Bougain le samedi 27 juin 2015 à 00h04

vendredi 26 juin 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 31 mai 2015

Mises à jour correctives 9.4.3, 9.3.8, 9.2.12, 9.1.17 et 9.0.21 bientôt disponibles. Préparez le déploiement : https://wiki.postgresql.org/wiki/May_2015_Fsync_Permissions_Bug

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20150531225635.GB8579@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Bruce Momjian pushed:

Heikki Linnakangas pushed:

Andrew Dunstan pushed:

Ãlvaro Herrera pushed:

  • Update README.tuplock. Multixact truncation is now handled differently, and this file hadn't gotten the memo. Per note from Amit Langote. I didn't use his patch, though. Also update the description of infomask bits, which weren't completely up to date either. This commit also propagates b01a4f6838 back to 9.3 and 9.4, which apparently I failed to do back then. http://git.postgresql.org/pg/commitdiff/cdbdc4382743fcfabb3437ea7c4d103adaa01324

Tom Lane pushed:

  • Explain CHECK constraint handling in postgres_fdw's IMPORT FOREIGN SCHEMA. The existing documentation could easily be misinterpreted, and it failed to explain the inconsistent-evaluation hazard that deterred us from supporting automatic importing of check constraints. Revise it. Etsuro Fujita, further expanded by me http://git.postgresql.org/pg/commitdiff/e70ec8230a2c0e7363bb7abf4d55dddbdec89fe1
  • Fix valgrind's "unaddressable bytes" whining about BRIN code. brin_form_tuple calculated an exact tuple size, then palloc'd and filled just that much. Later, brin_doinsert or brin_doupdate would MAXALIGN the tuple size and tell PageAddItem that that was the size of the tuple to insert. If the original tuple size wasn't a multiple of MAXALIGN, the net result would be that PageAddItem would memcpy a few more bytes than the palloc request had been for. AFAICS, this is totally harmless in the real world: the error is a read overrun not a write overrun, and palloc would certainly have rounded the request up to a MAXALIGN multiple internally, so there's no chance of the memcpy fetching off the end of memory. Valgrind, however, is picky to the byte level not the MAXALIGN level. Fix it by pushing the MAXALIGN step back to brin_form_tuple. (The other possible source of tuples in this code, brin_form_placeholder_tuple, was already producing a MAXALIGN'd result.) In passing, be a bit more paranoid about internal allocations in brin_form_tuple. http://git.postgresql.org/pg/commitdiff/79f2b5d583e2e2a7ccd13e31d0e20a900c8f2f61
  • Suppress occasional failures in brin regression test. brin.sql included a call of brin_summarize_new_values(), and expected it to always report exactly 5 summarization events. This failed sometimes during parallel regression tests, as a consequence of the database-wide VACUUM in gist.sql getting there first. The most future-proof way to avoid variation in the test results is to forget about using brin_summarize_new_values() and just do a plain "VACUUM brintest", which will exercise the same code anyway. Having done that, there's no need for preventing autovacuum on brintest; doing so just reduces the scope of test coverage, so let's not. http://git.postgresql.org/pg/commitdiff/1f303fd1be51f26553e7c95d8696aa4e28ece1c6
  • Remove configure check prohibiting threaded libpython on OpenBSD. According to recent tests, this case now works fine, so there's no reason to reject it anymore. (Even if there are still some OpenBSD platforms in the wild where it doesn't work, removing the check won't break any case that worked before.) We can actually remove the entire test that discovers whether libpython is threaded, since without the OpenBSD case there's no need to know that at all. Per report from Davin Potts. Back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/86832eb8912b9cac0f4961facb9efda81828e214
  • Fix portability issue in isolationtester grammar. specparse.y and specscanner.l used "string" as a token name. Now, bison likes to define each token name as a macro for the token code it assigns, which means those names are basically off-limits for any other use within the grammar file or included headers. So names as generic as "string" are dangerous. This is what was causing the recent failures on protosciurus: some versions of Solaris' sys/kstat.h use "string" as a field name. With late-model bison we don't see this problem because the token macros aren't defined till later (that is why castoroides didn't show the problem even though it's on the same machine). But protosciurus uses bison 1.875 which defines the token macros up front. This land mine has been there from day one; we'd have found it sooner except that protosciurus wasn't trying to run the isolation tests till recently. To fix, rename the token to "string_literal" which is hopefully less likely to collide with names used by system headers. Back-patch to all branches containing the isolation tests. http://git.postgresql.org/pg/commitdiff/aa9eac45ea868e6ddabc4eb076d18be10ce84c6a
  • Fix pg_get_functiondef() to print a function's LEAKPROOF property. Seems to have been an oversight in the original leakproofness patch. Per report and patch from Jeevan Chalke. In passing, prettify some awkward leakproof-related code in AlterFunction. http://git.postgresql.org/pg/commitdiff/f46edf479e2468a08caca2a03ec7e258930a7161
  • Fix assorted inconsistencies in our calls of readlink(). Ensure that we null-terminate the result string (one place in pg_rewind). Be paranoid about out-of-range results from readlink() (should not happen, but there is no good reason for some call sites to be careful about it and others not). Consistently use the whole buffer, not sometimes one byte less. Ensure we emit an appropriate errcode() in all cases. Spell the error messages the same way. The only serious bug here is the missing null-termination in pg_rewind, which is new code, so no need for a back-patch. Abhijit Menon-Sen and Tom Lane http://git.postgresql.org/pg/commitdiff/32f628be74d8ab43423ca7913b81f7eb21b312d4
  • Fix pg_rewind's handling of top-level symlinks. The previous coding suffered a null-pointer dereference if it found any symlink at the top level of $PGDATA. Fix that, and teach it to recurse into a symlink for pg_xlog, but not anything else. Per note from Abhijit Menon-Sen. http://git.postgresql.org/pg/commitdiff/0381fefaa44f04e17dffb7e46e7677374a4fb2c7
  • Fix fsync-at-startup code to not treat errors as fatal. Commit 2ce439f3379aed857517c8ce207485655000fc8e introduced a rather serious regression, namely that if its scan of the data directory came across any un-fsync-able files, it would fail and thereby prevent database startup. Worse yet, symlinks to such files also caused the problem, which meant that crash restart was guaranteed to fail on certain common installations such as older Debian. After discussion, we agreed that (1) failure to start is worse than any consequence of not fsync'ing is likely to be, therefore treat all errors in this code as nonfatal; (2) we should not chase symlinks other than those that are expected to exist, namely pg_xlog/ and tablespace links under pg_tblspc/. The latter restriction avoids possibly fsync'ing a much larger part of the filesystem than intended, if the user has left random symlinks hanging about in the data directory. This commit takes care of that and also does some code beautification, mainly moving the relevant code into fd.c, which seems a much better place for it than xlog.c, and making sure that the conditional compilation for the pre_sync_fname pass has something to do with whether pg_flush_data works. I also relocated the call site in xlog.c down a few lines; it seems a bit silly to be doing this before ValidateXLOGDirectoryStructure(). The similar logic in initdb.c ought to be made to match this, but that change is noncritical and will be dealt with separately. Back-patch to all active branches, like the prior commit. Abhijit Menon-Sen and Tom Lane http://git.postgresql.org/pg/commitdiff/d8179b001ae574da00c8f4549577095bf90f3337
  • Revert exporting of internal GUC variable "data_directory". This undoes a poorly-thought-out choice in commit 970a18687f9b3058, namely to export guc.c's internal variable data_directory. The authoritative variable so far as C code is concerned is DataDir; there is no reason for anything except specific bits of GUC code to look at the GUC variable. After yesterday's commits fixing the fsync-on-restart patch, the only remaining misuse of data_directory was in AlterSystemSetConfigFile(), which would be much better off just using a relative path anyhow: it's less code and it doesn't break if the DBA moves the data directory of a running system, which is a case we've taken some pains over in the past. This is mostly cosmetic, so no need for a back-patch (and I'd be hesitant to remove a global variable in stable branches anyway). http://git.postgresql.org/pg/commitdiff/da33a3894e0fc440ac53cdc0f2e360e703b13b8c
  • Check that all aliases of a built-in function have same leakproof property. opr_sanity.sql has a test checking that relevant properties of built-in functions match when the same C function is referenced by multiple pg_proc entries. The test neglected to check proleakproof, though, and when I added that condition it exposed that xideqint4 hadn't been updated to match xideq. So fix that as well, and in consequence bump catversion. This isn't very critical, so no need to worry about fixing back branches. http://git.postgresql.org/pg/commitdiff/1c8c656b3c217aaffc503ad703dcc60cdabe7445
  • Remove special cases for ETXTBSY from new fsync'ing logic. The argument that this is a sufficiently-expected case to be silently ignored seems pretty thin. Andres had brought it up back when we were still considering that most fsync failures should be hard errors, and it probably would be legit not to fail hard for ETXTBSY --- but the same is true for EROFS and other cases, which is why we gave up on hard failures. ETXTBSY is surely not a normal case, so logging the failure seems fine from here. http://git.postgresql.org/pg/commitdiff/57e1138bcc621ffeb8b1f1379ac4016a5c34d43e
  • initdb -S should now have an explicit check that $PGDATA is valid. The fsync code from the backend essentially assumes that somebody's already validated PGDATA, at least to the extent of it being a readable directory. That's safe enough for initdb's normal code path too, but "initdb -S" doesn't have any other processing at all that touches the target directory. To have reasonable error-case behavior, add a pg_check_dir call. Per gripe from Peter E. http://git.postgresql.org/pg/commitdiff/1943c000b7a22d3ca334196cfe3f7b8159b210c2
  • Adjust initdb to also not consider fsync'ing failures fatal. Make initdb's version of this logic look as much like the backend's as possible. This is much less critical than in the backend since not so many people use "initdb -S", but we want the same corner-case error handling in both cases. Back-patch to 9.3 where initdb -S option was introduced. Before that, initdb only had to deal with freshly-created data directories, wherein no failures should be expected. Abhijit Menon-Sen http://git.postgresql.org/pg/commitdiff/c07d8c963e39980f192e8daca73b7585ef76cc9b

Stephen Frost pushed:

Peter Eisentraut pushed:

Correctifs rejetés (à ce jour)

No one was disappointed this week :-)

Correctifs en attente

Peter Geoghegan sent in a patch to point out that for foreign tables, AddForeignUpdateTargets is not called INSERT operations with an ON CONFLICT DO UPDATE clause because that clause is not yet supported on foreign tables.

Peter Geoghegan sent in a flock of patches intended to ensure that some of the ON CONFLICT UPDATE always has its memory freed by its caller.

Emre Hasegeli sent in a patch to ensure that adjacent strategy numbers of range types match the central definitions.

Tomas Vondra sent in another revision of a patch to implement multivariate statistics.

Michael Paquier sent in another revision of a patch to add support for TAP tests on Windows.

Kaigai Kouhei sent in another revision of a patch to add a custom child jointree.

Uriy Zhuravlev sent in another revision of a patch to implement ALTER OPERATOR.

Uriy Zhuravlev sent in a patch to to add selectivity functions for intarray.

Andrew Dunstan sent in a patch to add a jsonb_set() function.

Tatsuo Ishii sent in a patch to fix some missing parts of the psql po translation for the Japanese language.

Peter Eisentraut sent in a WIP candidate candidate configuration which might eventually replace pgindent with clang-format.

SAWADA Masahiko sent in another revision of a patch to add a "frozen" bit into the visibility map.

Shigeru HANADA sent in another revision of a patch to implement postgres_fdw join pushdown.

boix sent in two revisions of a patch to use an ant colony optimization for GEQO.

Michael Paquier sent in another revision of a doc patch to better describe the behavior of txid_current().

Michael Paquier sent in a patch to fix a memory leak in XLogFileCopy.

Craig Ringer sent in a patch to document that directly callable functions may use fn_extra.

Naoya Anzai sent in a patch to add more vacuum statistics.

Abhijit Menon-Sen sent in a patch to ensure that S_IRUSR|S_IWUSR are set only with O_CREAT.

Peter Eisentraut sent in a patch to allow ALTER TABLE ENABLE TRIGGER to work on VIEWs.

Thomas Munro and Robert Haas traded patches to tolerate missing offset segments.

Tomas Vondra and Tom Lane traded patches to fix cost estimates for nested loop semijoin.

Andreas Karlsson sent in a patch to reload SSL certificates on SIGHUP, which makes it possible to enable or disable SSL entirely without a restart.

Andreas Karlsson sent in a patch to fix autoconf deprecation warnings.

par N Bougain le vendredi 26 juin 2015 à 23h35

mardi 16 juin 2015

Guillaume Lelarge

Différences entre les versions beta du livre

Je me suis rendu compte ce week-end qu'on n'avait pas publié d'informations sur ce qui avait été ajouté entre les différentes versions beta, en dehors des nouveaux chapitres. Voici donc la liste des modifications, un peu éditée pour être plus lisible :

Pour la beta2 :

  • Nouveaux chapitres
    • protocole de communication
    • connexions
  • Chapitre fichiers
    • ajout d'une note sur l'option --no-clean de la commande initdb
    • ajout d'une note sur les versions 9.1 (et inférieures) et la colonne spclocation du catalogue pg_tablespace
    • ajout d'une note sur le fichier pgstat.stat des versions 9.3 et antérieures
  • Chapitre processus
    • refonte des sections pour trier les processus par activité (et non par nom)
    • revue du résumé sur les processus d'écriture dans les fichiers de données suite à une remarque d'un lecteur dans le forum du livre (forum uniquement accessible par les lecteurs actuels)
    • correction des processus équivalents au niveau Oracle, suite là-aussi à un autre commentaire d'un lecteur
  • Chapitre mémoire
    • ajout de deux paragraphes sur l'utilisation (partielle) du cache pour les requêtes
    • présentation de deux colonnes de la vue pg_stat_database permettant de quantifier l'utilisation des fichiers temporaires

Et pour la beta 3 :

  • Nouveaux chapitres
    • gestion des objets
    • transactions
  • Global
    • remplacement du terme maître par serveur primaire et du terme esclave par serveur secondaire (suite à la demande d'un lecteur)
  • Chapitre mémoire
    • ajout d'une note sur l'intérêt du paramètre maintenance_work_mem dans le cadre d'un import de données avec pg_restore
  • Chapitre processus
    • ajout d'une partie sur les écritures dans les fichiers de données suite à un CHECKPOINT, avec quelques graphes, pour mieux expliquer les différents paramètres checkpoint_*
  • Chapitre Fichiers
    • ajout d'un paragraphe sur la génération des relfilnode
    • ajout d'infos sur le LSN et les journaux de transactions

Je publierais dans ce blog les nouveautés de chaque version beta à chaque sortie, ce sera plus facile pour les lecteurs.

par Guillaume Lelarge le mardi 16 juin 2015 à 20h32

vendredi 12 juin 2015

Actualités PostgreSQL.fr

Nouvelles versions mineures au 12 juin 2015

Le PostgreSQL Global Development Group a publié une mise à jour pour toutes les versions supportées du SGBD, soit les versions 9.4.4, 9.3.9, 9.2.13, 9.1.18 et 9.0.22.
Cette mise à jour a pour but principal de corriger les bogues qui n'ont pas été corrigés par les mises à jour précédentes.

Il est impératif de les appliquer immédiatement pour les utilisateurs ayant déjà appliqué les mises à jour de mai et juin.
Les autres utilisateurs pourront attendre le prochain arrêt de service programmé.

Correctifs des récupérations en cas de panne

Les mises à jour précédentes ont tenté de corriger un problème de "multixact wraparound" (ou bouclage d'identifiant de transactions multiples) sur les versions 9.3 et 9.4 de PostgreSQL.
Mais elles n'ont pas corrigé le nettoyage des transactions multiples lors de récupération apès crash. Cela peut empêcher un serveur de redémarrer après un crash.
C'est pourquoi il est recommandé aux utilisateurs de ces versions de procéder à la mise à jour le plus vite possible.

Les serveurs précédemment passés en version 9.3 avec pg_upgrade, ou ceux qui tournent en 9.4, suite à une autre mise à jour, peuvent voir tourner un autovacuum sur l'ensemble des tables immédiatement après l'application de l'actuelle mise à jour.
Si les bases sont volumineuses, il peut être plus judicieux de procéder à un VACUUM manuel, avant la mise à jour, pour maîtriser les performances suite à cette maintenance.

N'hésitez pas à consulter les notes de versions pour les détails ( Notes de version ).

Autres correctifs et améliorations

En plus des correctifs précédents, quelques autres correctifs ont été appliqués à cette version.

On trouve :

 * le correctif des rares cas ou l'invalidation du cache des relations n'était pas effectué ;
 * la suppression des verrous morts entre de nouvelles sessions et un CREATE/DROP DATABASE ;
 * l'amélioration de la planification des requêtes pour les semi-jointures et anti-jointures.

Mises à jour cumulatives

Toutes les mises à jour de PostgreSQL sont cumulatives. Comme cette mise à jour fixe un certain nombre de problèmes malencontreusement apportées par les mises à jour précédentes, il est préférable d'installer directement cette mise à jour plutôt que les précédentes.

Comme cette mise à jour clôt plusieurs bogues de gestion des transactions multiples, le projet PostgreSQL ne prévoit pas de nouvelles mises à jour prochainement.

Application de la mise à jour

Comme pour les autres mises à jour, il n'est pas nécessaire de procéder à l'export et au réimport des données, ou d'utiliser pg_upgrade. Il suffit d'installer les nouveaux binaires et de redémarrer PostgreSQL.

Au cas où des mises à jour mineures auraient été omises, il peut y avoir des opérations suppplémentaires à effectuer. Toutes les informations se trouvent dans les notes de versions. Voir également les informations complémentaires en cas de passage à la version 9.3 avec pg_upgrade.

Liens :

 * Téléchargement
 * Notes de version

par SAS le vendredi 12 juin 2015 à 14h35

vendredi 29 mai 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 24 mai 2015

Les mises à jour de sécurité et de correction 9.4.2, 9.3.7, 9.2.11, 9.1.16 et 9.0.20 sont disponibles. Déployez au plus tôt : http://www.postgresql.org/about/news/1587/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Use += not = to set makefile variables after including base makefiles. The previous coding in hstore_plpython and ltree_plpython wiped out any values set by the base makefiles. This at least had the effect of running the tests in "regression" not "contrib_regression" as expected. These being pretty new modules, there might be other bad effects we'd not noticed yet. http://git.postgresql.org/pg/commitdiff/b14cf229f4bd7238be2e31d873dc5dd241d3871e
  • Fix failure to copy IndexScan.indexorderbyops in copyfuncs.c. This oversight results in a crash at executor startup if the plan has been copied. outfuncs.c was missed as well. While we could probably have taught both those files to cope with the originally chosen representation of an Oid array, it would have been painful, not least because there'd be no easy way to verify the array length. An Oid List is far easier to work with. And AFAICS, there is no particular notational benefit to using an array rather than a list in the existing parts of the patch either. So just change it to a list. Error in commit 35fcb1b3d038a501f3f4c87c05630095abaaadab, which is new, so no need for back-patch. http://git.postgresql.org/pg/commitdiff/424661913c06af76a46fdff9cc24cc57abf14fb3
  • Recognize "REGRESS_OPTS += ..." syntax in MSVC build scripts. Necessitated by commit b14cf229f4bd7238be2e31d873dc5dd241d3871e. Per buildfarm. http://git.postgresql.org/pg/commitdiff/f5916bb7b53f8a77c95c00c5b287659958891178
  • Put back a backwards-compatible version of sampling support functions. Commit 83e176ec18d2a91dbea1d0d1bd94c38dc47cd77c removed the longstanding support functions for block sampling without any consideration of the impact this would have on third-party FDWs. The new API is not notably more functional for FDWs than the old, so forcing them to change doesn't seem like a good thing. We can provide the old API as a wrapper (more or less) around the new one for a minimal amount of extra code. http://git.postgresql.org/pg/commitdiff/4db485e75b9672126963ae4052b50f473b30a097
  • Revert "Change pg_seclabel.provider and pg_shseclabel.provider to type "name"." This reverts commit b82a7be603f1811a0a707b53c62de6d5d9431740. There is a better (less invasive) way to fix it, which I will commit next. http://git.postgresql.org/pg/commitdiff/afee04352bc01b79cde33c018a82c2eeb1ce84eb
  • Avoid collation dependence in indexes of system catalogs. No index in template0 should have collation-dependent ordering, especially not indexes on shared catalogs. For most textual columns we avoid this issue by using type "name" (which sorts per strcmp()). However there are a few indexed columns that we'd prefer to use "text" for, and for that, the default opclass text_ops is unsafe. Fortunately, text_pattern_ops is safe (it sorts per memcmp()), and it has no real functional disadvantage for our purposes. So change the indexes on pg_seclabel.provider and pg_shseclabel.provider to use text_pattern_ops. In passing, also mark pg_replication_origin.roname as using text_pattern_ops --- for some reason it was labeled varchar_pattern_ops which is just wrong, even though it accidentally worked. Add regression test queries to catch future errors of these kinds. We still can't do anything about the misdeclared pg_seclabel and pg_shseclabel indexes in back branches
  • http://git.postgresql.org/pg/commitdiff/0b28ea79c044a0d3779081dc909a6dc0ce93b991
  • - Change pg_seclabel.provider and pg_shseclabel.provider to type "name". These were "text", but that's a bad idea because it has collation-dependent ordering. No index in template0 should have collation-dependent ordering, especially not indexes on shared catalogs. There was general agreement that provider names don't need to be longer than other identifiers, so we can fix this at a small waste of table space by changing from text to name. There's no way to fix the problem in the back branches, but we can hope that security labels don't yet have widespread-enough usage to make it urgent to fix. There needs to be a regression sanity test to prevent us from making this same mistake again; but before putting that in, we'll need to get rid of similar brain fade in the recently-added pg_replication_origin catalog. Note: for lack of a suitable testing environment, I've not really exercised this change. I trust the buildfarm will show up any mistakes. http://git.postgresql.org/pg/commitdiff/b82a7be603f1811a0a707b53c62de6d5d9431740
  • Another typo fix. In the spirit of the season. http://git.postgresql.org/pg/commitdiff/a6a66bd647d471aeb55d8ba3e24d197ccd8a5abb
  • Improve packing/alignment annotation for ItemPointerData. We want this struct to be exactly a series of 3 int16 words, no more and no less. Historically, at least, some ARM compilers preferred to pad it to 8 bytes unless coerced. Our old way of doing that was just to use __attribute__((packed)), but as pointed out by Piotr Stefaniak, that does too much: it also licenses the compiler to give the struct only byte-alignment. We don't want that because it adds access overhead, possibly quite significant overhead. According to the GCC manual, what we want requires also specifying __attribute__((align(2))). It's not entirely clear if all the relevant compilers accept this pragma as well, but we can hope the buildfarm will tell us if not. We can also add a static assertion that should fire if the compiler padded the struct. Since the combination of these pragmas should define exactly what we want on any compiler that accepts them, let's try using them wherever we think they exist, not only for __arm__. (This is likely to expose that the conditional definitions in c.h are inadequate, but finding that out would be a good thing.) The immediate motivation for this is that the current definition of ExecRowMark allows its curCtid field to be misaligned. It is not clear whether there are any other uses of ItemPointerData with a similar hazard. We could change the definition of ExecRowMark if this doesn't work, but it would be far better to have a future-proof fix. Piotr Stefaniak, some further hacking by me http://git.postgresql.org/pg/commitdiff/d4b538ea367de43b2f2b939621272682417cd290
  • More fixes for lossy-GiST-distance-functions patch. Paul Ramsey reported that commit 35fcb1b3d038a501f3f4c87c05630095abaaadab induced a core dump on commuted ORDER BY expressions, because it was assuming that the indexorderby expression could be found verbatim in the relevant equivalence class, but it wasn't there. We really don't need anything that complicated anyway; for the data types likely to be used for index ORDER BY operators in the foreseeable future, the exprType() of the ORDER BY expression will serve fine. (The case where we'd have to work harder is where the ORDER BY expression's result is only binary-compatible with the declared input type of the ordering operator; long before worrying about that, one would need to get rid of GiST's hard-wired assumption that said datatype is float8.) Aside from fixing that crash and adding a regression test for the case, I did some desultory code review: nodeIndexscan.c was likewise overthinking how hard it ought to work to identify the datatype of the ORDER BY expressions. Add comments explaining how come nodeIndexscan.c can get away with simplifying assumptions about NULLS LAST ordering and no backward scan. Revert no-longer-needed changes of find_ec_member_for_tle(); while the new definition was no worse than the old, it wasn't better either, and it might cause back-patching pain. Revert entirely bogus additions to genam.h. http://git.postgresql.org/pg/commitdiff/c5dd8ead403f85bd041590d2e3e79b72830472d4
  • Fix recently-introduced crash in array_contain_compare(). Silly oversight in commit 1dc5ebc9077ab742079ce5dac9a6664248d42916: when array2 is an expanded array, it might have array2->xpn.dnulls equal to NULL, indicating the array is known null-free. The code wasn't expecting that, because it formerly always used deconstruct_array() which always delivers a nulls array. Per bug #13334 from Regina Obe. http://git.postgresql.org/pg/commitdiff/49ad32d5d99cb4a79bf648c0b7f9eca19b54cf1d
  • Last-minute updates for release notes. Add entries for security issues. Security: CVE-2015-3165 through CVE-2015-3167 http://git.postgresql.org/pg/commitdiff/19d47ed2da1e4d08ffab7e8ba1b1c4c614e7f296
  • Last-minute updates for release notes. Revise description of CVE-2015-3166, in line with scaled-back patch. Change release date. Security: CVE-2015-3166 http://git.postgresql.org/pg/commitdiff/5cb8519ceb62516636362a7e8e06b99b3e1bf138
  • Revert error-throwing wrappers for the printf family of functions. This reverts commit 16304a013432931e61e623c8d85e9fe24709d9ba, except for its changes in src/port/snprintf.c; as well as commit cac18a76bb6b08f1ecc2a85e46c9d2ab82dd9d23 which is no longer needed. Fujii Masao reported that the previous commit caused failures in psql on OS X, since if one exits the pager program early while viewing a query result, psql sees an EPIPE error from fprintf --- and the wrapper function thought that was reason to panic. (It's a bit surprising that the same does not happen on Linux.) Further discussion among the security list concluded that the risk of other such failures was far too great, and that the one-size-fits-all approach to error handling embodied in the previous patch is unlikely to be workable. This leaves us again exposed to the possibility of the type of failure envisioned in CVE-2015-3166. However, that failure mode is strictly hypothetical at this point: there is no concrete reason to believe that an attacker could trigger information disclosure through the supposed mechanism. In the first place, the attack surface is fairly limited, since so much of what the backend does with format strings goes through stringinfo.c or psprintf(), and those already had adequate defenses. In the second place, even granting that an unprivileged attacker could control the occurrence of ENOMEM with some precision, it's a stretch to believe that he could induce it just where the target buffer contains some valuable information. So we concluded that the risk of non-hypothetical problems induced by the patch greatly outweighs the security risks. We will therefore revert, and instead undertake closer analysis to identify specific calls that may need hardening, rather than attempt a universal solution. We have kept the portion of the previous patch that improved snprintf.c's handling of errors when it calls the platform's sprintf(). That seems to be an unalloyed improvement. Security: CVE-2015-3166 http://git.postgresql.org/pg/commitdiff/0c071936e94c6859afb2ec8d2c8dddf7bcdab7ee
  • Still more fixes for lossy-GiST-distance-functions patch. Fix confusion in documentation, substantial memory leakage if float8 or float4 are pass-by-reference, and assorted comments that were obsoleted by commit 98edd617f3b62a02cb2df9b418fcc4ece45c7ec0. http://git.postgresql.org/pg/commitdiff/821b821a2421beaa58225ff000833df69fb962c5
  • Fix incorrect snprintf() limit. Typo in commit 7cbee7c0a. No practical effect since the buffer should never actually be overrun, but various compilers and static analyzers will whine about it. Petr Jelinek http://git.postgresql.org/pg/commitdiff/72809480d658fbc0654239b2f089991c077c676a
  • Add error check for lossy distance functions in index-only scans. Maybe we should actually support this, but for the moment let's just throw an error if the opclass tries it. http://git.postgresql.org/pg/commitdiff/f84c8601d604811a530dadb53ddb52f08639e72b
  • Remove no-longer-required function declarations. Remove a bunch of "extern Datum foo(PG_FUNCTION_ARGS);" declarations that are no longer needed now that PG_FUNCTION_INFO_V1(foo) provides that. Some of these were evidently missed in commit e7128e8dbb305059, but others were cargo-culted in in code added since then. Possibly that can be blamed in part on the fact that we'd not fixed relevant documentation examples, which I've now done. http://git.postgresql.org/pg/commitdiff/91e79260f636ab4d5a43910b6a38bc75651ad14c
  • Add a bit more commentary about regex's colormap tree data structure. Per an off-list question from Piotr Stefaniak. http://git.postgresql.org/pg/commitdiff/23116d5437d0e8d077e7fd5391f5fa0fc781b7d2
  • Rename pg_shdepend.c's typedef "objectType" to SharedDependencyObjectType. The name objectType is widely used as a field name, and it's pure luck that this conflict has not caused pgindent to go crazy before. It messed up pg_audit.c pretty good though. Since pg_shdepend.c doesn't export this typedef and only uses it in three places, changing that seems saner than changing the field usages. Back-patch because we're contemplating using the union of all branch typedefs for future pgindent runs, so this won't fix anything if it stays the same in back branches. http://git.postgresql.org/pg/commitdiff/17b48a1a9f87f7479d38dcc78a27c23f1f8124f8
  • Manual cleanup of pgindent results. Fix some places where pgindent did silly stuff, often because project style wasn't followed to begin with. (I've not touched the atomics headers, though.) http://git.postgresql.org/pg/commitdiff/2aa0476dc38f7e510b8cde627e83b4c76fa05d61

Peter Eisentraut a poussé :

Fujii Masao a poussé :

Heikki Linnakangas a poussé :

  • Put back stats-collector restarting code, removed accidentally. Removed that code snippet accidentally in the archive_mode='always' patch. Also, use varname-tags for archive_command in the docs. Fujii Masao http://git.postgresql.org/pg/commitdiff/4df132895016c6a99355776a8df284ff011a2e94
  • Fix typo in comment. Jim Nasby http://git.postgresql.org/pg/commitdiff/8cc7a4c5fdbe43b9b16b4cf3e07c8115107a8d4e
  • Fix off-by-one error in Assertion. The point of the assertion is to ensure that the arrays allocated in stack are large enough, but the check was one item short. This won't matter in practice because MaxIndexTuplesPerPage is an overestimate, so you can't have that many items on a page in reality. But let's be tidy. Spotted by Anastasia Lubennikova. Backpatch to all supported versions, like the patch that added the assertion. http://git.postgresql.org/pg/commitdiff/b48437d11b9389d724c037385a5cae824d4f8049
  • Collection of typo fixes. Use "a" and "an" correctly, mostly in comments. Two error messages were also fixed (they were just elogs, so no translation work required). Two function comments in pg_proc.h were also fixed. Etsuro Fujita reported one of these, but I found a lot more with grep. Also fix a few other typos spotted while grepping for the a/an typos. For example, "consists out of ..." -> "consists of ...". Plus a "though"/ "through" mixup reported by Euler Taveira. Many of these typos were in old code, which would be nice to backpatch to make future backpatching easier. But much of the code was new, and I didn't feel like crafting separate patches for each branch. So no backpatching. http://git.postgresql.org/pg/commitdiff/4fc72cc7bb9d2105261b8ee45558af50d788cd19
  • Fix more typos in comments. Patch by CharSyam, plus a few more I spotted with grep. http://git.postgresql.org/pg/commitdiff/fa60fb63e511e7bbcf57ce972338711593a5e7c9
  • At promotion, don't leave behind a partial segment on the old timeline. With commit de768844, a copy of the partial segment was archived with the .partial suffix, but the original file was still left in pg_xlog, so it didn't actually solve the problems with archiving the partial segment that it was supposed to solve. With this patch, the partial segment is renamed rather than copied, so we only archive it with the .partial suffix. Also be more robust in detecting if the last segment is already being archived. Previously I used XLogArchiveIsBusy() for that, but that's not quite right. With archive_mode='always', there might be a .ready file for it, and we don't want to rename it to .partial in that case. The old segment is needed until we're fully committed to the new timeline, i.e. until we've written the end-of-recovery WAL record and updated the min recovery point and timeline in the control file. So move the renaming later in the startup sequence, after all that's been done. http://git.postgresql.org/pg/commitdiff/7cbee7c0a1db668c60c020a3fd1e3234daa562a9

Noah Misch a poussé :

  • Permit use of vsprintf() in PostgreSQL code. The next commit needs it. Back-patch to 9.0 (all supported versions). http://git.postgresql.org/pg/commitdiff/cac18a76bb6b08f1ecc2a85e46c9d2ab82dd9d23
  • Check return values of sensitive system library calls. PostgreSQL already checked the vast majority of these, missing this handful that nearly cannot fail. If putenv() failed with ENOMEM in pg_GSS_recvauth(), authentication would proceed with the wrong keytab file. If strftime() returned zero in cache_locale_time(), using the unspecified buffer contents could lead to information exposure or a crash. Back-patch to 9.0 (all supported versions). Other unchecked calls to these functions, especially those in frontend code, pose negligible security concern. This patch does not address them. Nonetheless, it is always better to check return values whose specification provides for indicating an error. In passing, fix an off-by-one error in strftime_win32()'s invocation of WideCharToMultiByte(). Upon retrieving a value of exactly MAX_L10N_DATA bytes, strftime_win32() would overrun the caller's buffer by one byte. MAX_L10N_DATA is chosen to exceed the length of every possible value, so the vulnerable scenario probably does not arise. Security: CVE-2015-3166 http://git.postgresql.org/pg/commitdiff/fd97bd411d1da45b79e63c2124741f8e82cc5a5c
  • Prevent a double free by not reentering be_tls_close(). Reentering this function with the right timing caused a double free, typically crashing the backend. By synchronizing a disconnection with the authentication timeout, an unauthenticated attacker could achieve this somewhat consistently. Call be_tls_close() solely from within proc_exit_prepare(). Back-patch to 9.0 (all supported versions). Benkocs Norbert Attila Security: CVE-2015-3165 http://git.postgresql.org/pg/commitdiff/b0ce385032d72d6acf1e330f733013553fe6affe
  • pgcrypto: Report errant decryption as "Wrong key or corrupt data". This has been the predominant outcome. When the output of decrypting with a wrong key coincidentally resembled an OpenPGP packet header, pgcrypto could instead report "Corrupt data", "Not text data" or "Unsupported compression algorithm". The distinct "Corrupt data" message added no value. The latter two error messages misled when the decrypted payload also exhibited fundamental integrity problems. Worse, error message variance in other systems has enabled cryptologic attacks; see RFC 4880 section "14. Security Considerations". Whether these pgcrypto behaviors are likewise exploitable is unknown. In passing, document that pgcrypto does not resist side-channel attacks. Back-patch to 9.0 (all supported versions). Security: CVE-2015-3167 http://git.postgresql.org/pg/commitdiff/85270ac7a24a50d43ba4bd4d7af1e28b14dee7ee
  • Add error-throwing wrappers for the printf family of functions. All known standard library implementations of these functions can fail with ENOMEM. A caller neglecting to check for failure would experience missing output, information exposure, or a crash. Check return values within wrappers and code, currently just snprintf.c, that bypasses the wrappers. The wrappers do not return after an error, so their callers need not check. Back-patch to 9.0 (all supported versions). Popular free software standard library implementations do take pains to bypass malloc() in simple cases, but they risk ENOMEM for floating point numbers, positional arguments, large field widths, and large precisions. No specification demands such caution, so this commit regards every call to a printf family function as a potential threat. Injecting the wrappers implicitly is a compromise between patch scope and design goals. I would prefer to edit each call site to name a wrapper explicitly. libpq and the ECPG libraries would, ideally, convey errors to the caller rather than abort(). All that would be painfully invasive for a back-patched security fix, hence this compromise. Security: CVE-2015-3166 http://git.postgresql.org/pg/commitdiff/16304a013432931e61e623c8d85e9fe24709d9ba

Robert Haas a poussé :

Andres Freund a poussé :

  • Attach ON CONFLICT SET ... WHERE to the correct planstate. The previous coding was a leftover from attempting to hang all the on conflict logic onto modify table's child nodes. It appears to not have actually caused problems except for explain. Add test exercising the broken and some other code paths. Author: Peter Geoghegan and Andres Freund http://git.postgresql.org/pg/commitdiff/e4942f7a56efcfaabed5db7bde29ee21bef2f6e2
  • Various fixes around ON CONFLICT for rule deparsing. Neither the deparsing of the new alias for INSERT's target table, nor of the inference clause was supported. Also fixup a typo in an error message. Add regression tests to test those code paths. Author: Peter Geoghegan http://git.postgresql.org/pg/commitdiff/9bc77c45199c7d2e525cd5b1457d5a57f6e9edb0
  • Refactor ON CONFLICT index inference parse tree representation. Defer lookup of opfamily and input type of a of a user specified opclass until the optimizer selects among available unique indexes; and store the opclass in the parse analyzed tree instead. The primary reason for doing this is that for rule deparsing it's easier to use the opclass than the previous representation. While at it also rename a variable in the inference code to better fit it's purpose. This is separate from the actual fixes for deparsing to make review easier. http://git.postgresql.org/pg/commitdiff/0740cbd7593d871858c352fab29a59cf7fa54b00
  • Remove the new UPSERT command tag and use INSERT instead. Previously, INSERT with ON CONFLICT DO UPDATE specified used a new command tag -- UPSERT. It was introduced out of concern that INSERT as a command tag would be a misrepresentation for ON CONFLICT DO UPDATE, as some affected rows may actually have been updated. Alvaro Herrera noticed that the implementation of that new command tag was incomplete; in subsequent discussion we concluded that having it doesn't provide benefits that are in line with the compatibility breaks it requires. Catversion bump due to the removal of PlannedStmt->isUpsert. Author: Peter Geoghegan Discussion: 20150520215816.GI5885@postgresql.org http://git.postgresql.org/pg/commitdiff/631d7490074cdaef8026db57a5f2772b8730f600
  • Fix yet another bug in ON CONFLICT rule deparsing. Expand testing of rule deparsing a good bit, it's evidently needed. Author: Peter Geoghegan, Andres Freund Discussion: CAM3SWZQmXxZhQC32QVEOTYfNXJBJ_Q2SDENL7BV14Cq-zL0FLg@mail.gmail.com http://git.postgresql.org/pg/commitdiff/284bef297733e553c73f1c858e0ce1532f754d18

Simon Riggs a poussé :

Andrew Dunstan a poussé :

  • Unpack jbvBinary objects passed to pushJsonbValue pushJsonbValue was accepting jbvBinary objects passed as WJB_ELEM or WJB_VALUE data. While this succeeded, when those objects were later encountered in attempting to convert the result to Jsonb, errors occurred. With this change we ghuarantee that a JSonbValue constructed from calls to pushJsonbValue does not contain any jbvBinary objects. This cures a problem observed with jsonb_delete. This means callers of pushJsonbValue no longer need to perform this unpacking themselves. A subsequent patch will perform some cleanup in that area. The error was not triggered by any 9.4 code, but this is a publicly visible routine, and so the error could be exercised by third party code, therefore backpatch to 9.4. Bug report from Peter Geoghegan, fix by me. http://git.postgresql.org/pg/commitdiff/5302760a50332a684e35b9865ff47ff5fd4970c2

Bruce Momjian a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Álvaro Herrera sent in another revision of a patch to fix some infelicities with MultiXact in recovery.
  • Uriy Zhuravlev sent in a patch to implement ALTER OPERATOR.
  • John Gorman sent in a patch to preserve errno across errmsg() calls.
  • Joshua Drake sent in a doc patch for backups.
  • Jim Nasby sent in another revision of a patch to add an optional parameter to the pg_*_backend functions which allows skipping the backend making the call.
  • Jeff Janes sent in a patch to fix max WAL size in recovery.
  • Alexander Shulgin sent in a patch to generalize the JSON-producing functions in utils/adt/json.c and to provide a set of callbacks which can be overridden the same way that is already provided for parsing JSON.
  • Andrew Gierth sent in a patch to change the type of GROUPING() from int4 to int8.
  • Robert Haas sent in another revision of a patch to allow assessing parallel safety.
  • Alexander Shulgin sent in another revision of a patch to add a --strict-include parameter to pg_dump.
  • Andrew Dunstan sent in a patch to rename jsonb_replace to jsonb_set with a boolean create_missing flag that defaults to false.
  • Amit Kapila sent in another revision of a patch to implement parallel seq scan.
  • Pavel Stehule sent in a patch to allow SET ROLE TO to tab-complete in psql, just as SET ROLE now does.
  • Fabrízio de Royes Mello sent in a patch to fix some spacing in the ALTER TABLE documentation.
  • CharSyam sent in a patch to change magic constants to DEFINE value for readability.
  • Noah Misch sent in a patch to fix some tar name issues.
  • Fabien COELHO sent in another revision of a patch to extend pgbench expressions with functions.
  • Petr Korobeinikov sent in another revision of a patch to add support for \ev viewname and \sv viewname to psql.

par N Bougain le vendredi 29 mai 2015 à 14h34

lundi 25 mai 2015

Dimitri Fontaine

PostgreSQL au JUG de Montpellier

J'ai eu le plaisir de présenter PostgreSQL au Java User Group de Montpellier le 20 mai dernier dans le cadre d'une soirée PostgreSQL is YeSQL. L'accueil réservé par le JUG a été parfait et je tiens à les remercier pour une très bonne soirée de partage et d'échanges autour de PostgreSQL.

Les slides des présentations sont disponibles. Nous avons commencé par une présentation générale sur comment et pourquoi utiliser PostgreSQL en 2015 :

Puis nous avons continué la soirée avec une présentation des Extensions PostgreSQL, dont ce blog regorge d'exemples :

Une série d'articles sont disponibles sur ce blog et reprennent en détails les éléments présentés dans cette deuxième partie :

 

Next month, Postgres Open 2014 is happening in Chicago, and I'll have the pleasure to host a tutorial about PostgreSQL Extensions Writing & Using Postgres Extensions, and a talk aimed at developers wanting to make the best out of PostgreSQL, PostgreSQL for developers:

 

PHP Tour 2014

June, 27 2014

En début de semaine se tenait le PHP Tour 2014 à Lyon, et j'ai eu le privilège d'y être invité afin de présenter comment Utiliser PostgreSQL en 2014.

 

Last week some PostgreSQL users, contributors and advocates have organized a really great conference in Stockholm, Sweden, where I had the please to give the following talk:

 

In our previous article Aggregating NBA data, PostgreSQL vs MongoDB we spent time comparing the pretty new MongoDB Aggregation Framework with the decades old SQL aggregates. Today, let's showcase more of those SQL aggregates, producing a nice histogram right from our SQL console.

 

When reading the article Crunching 30 Years of NBA Data with MongoDB Aggregation I coulnd't help but think that we've been enjoying aggregates in SQL for 3 or 4 decades already. When using PostgreSQL it's even easy to actually add your own aggregates given the SQL command create aggregate.

 

Back from the FODESM 2014 Conference, here's the slides I've been using for the Advanced Extension Use Cases talk I gave, based on the ongoing work to be found under the Tour of Extensions index in this web site.

 

Back From Dublin

November, 05 2013

Last week I had the pleasure to present two talks at the awesome PostgreSQL Conference Europe. The first one was actually a tutorial about Writing & using Postgres Extensions where we spent 3 hours on what are PostgreSQL Extensions, what you can expect from them, and how to develop a new one. Then I also had the opportunity to present the new version of pgloader in a talk about Migrating from MySQL to PostgreSQL.

 

Denormalizing Tags

October, 24 2013

In our Tour of Extensions today's article is about advanced tag indexing. We have a great data collection to play with and our goal today is to be able to quickly find data matching a complex set of tags. So, let's find out those lastfm tracks that are tagged as blues and rhythm and blues, for instance.

 

At the Open World Forum two weeks ago I had the pleasure to meet with Colin Charles. We had a nice talk about the current state of both MariaDB and PostgreSQL, and even were both interviewed by the Open World Forum Team. The interview is now available online. Dear French readers, it's in English.

 

PostgreSQL is an all round impressive Relational DataBase Management System which implements the SQL standard (see the very useful reference page Comparison of different SQL implementations for details). PostgreSQL also provides with unique solutions in the database market and has been leading innovation for some years now. Still, there's no support for Autonomous Transactions within the server itself. Let's have a look at how to easily implement them with PL/Proxy.

 

Let's get back to our Tour of Extensions that had to be kept aside for awhile with other concerns such as last chance PostgreSQL data recovery. Now that we have a data loading tool up to the task (read about it in the Loading Geolocation Data article) we're going to be able to play with the awesome ip4r extension from RhodiumToad.

 

Last Friday I had the chance to be speaking at the Open World Forum in the NewSQL track, where we had lots of interest and excitement around the NoSQL offerings. Of course, my talk was about explaining how PostgreSQL is Web Scale with some historical background and technical examples about what this database engine is currently capable of.

 

Using trigrams against typos

September, 06 2013

In our ongoing Tour of Extensions we played with earth distance in How far is the nearest pub? then with hstore in a series about trigger, first to generalize Trigger Parameters then to enable us to Auditing Changes with Hstore. Today we are going to work with pg_trgm which is the trigrams PostgreSQL extension: its usage got seriously enhanced in recent PostgreSQL releases and it's now a poor's man Full Text Search engine.

 

In a previous article about Trigger Parameters we have been using the extension hstore in order to compute some extra field in our records, where the fields used both for the computation and for storing the results were passed in as dynamic parameters. Today we're going to see another trigger use case for hstore: we are going to record changes made to our tuples.

 

Trigger Parameters

August, 23 2013

Sometimes you want to compute values automatically at INSERT time, like for example a duration column out of a start and an end column, both timestamptz. It's easy enough to do with a BEFORE TRIGGER on your table. What's more complex is to come up with a parametrized spelling of the trigger, where you can attach the same stored procedure to any table even when the column names are different from one another.

 

There was SQL before window functions and SQL after window functions: that's how powerful this tool is. Being that of a deal breaker unfortunately means that it can be quite hard to grasp the feature. This article aims at making it crystal clear so that you can begin using it today and are able to reason about it and recognize cases where you want to be using window functions.

 

In our recent article about The Most Popular Pub Names we did have a look at how to find the pubs nearby, but didn't compute the distance in between that pub and us. That's because how to compute a distance given a position on the earth expressed as longitude and latitude is not that easy. Today, we are going to solve that problem nonetheless, thanks to PostgreSQL Extensions.

 

In his article titled The Most Popular Pub Names Ross Lawley did show us how to perform some quite interesting geographic queries against MongoDB, using some nice Open Data found at the Open Street Map project.

 

In a recent article here we've been talking about how do do Batch Updates in a very efficient way, using the Writable CTE features available in PostgreSQL 9.1. I sometime read how Common Table Expressions changed the life of fellow DBAs and developers, and would say that Writable CTE are at least the same boost again.

 

In a recent article Craig Kerstiens from Heroku did demo the really useful crosstab extension. That function allows you to pivot a table so that you can see the data from different categories in separate columns in the same row rather than in separate rows. The article from Craig is Pivoting in Postgres.

 

Tonight I had the pleasure to present a talk at the Dublin PostgreSQL User Group using remote technologies. The talk is about how to make the most ouf of PostgreSQL when using SQL as a developer, and tries to convince you to dive into mastering SQL by showing how to solve an application example all in SQL, using window functions and common table expressions.

 

Nearest Big City

May, 02 2013

In this article, we want to find the town with the greatest number of inhabitants near a given location.

 

The Need For Speed

March, 29 2013

Hier se tenait la cinquième édition de la conférence organisée par dalibo, où des intervenants extérieurs sont régulièrement invités. Le thème hier était à la fois clair et très vaste : la performance.

 

Batch Update

March, 15 2013

Performance consulting involves some tricks that you have to teach over and over again. One of them is that SQL tends to be so much better at dealing with plenty of rows in a single statement when compared to running as many statements, each one against a single row.

 

HyperLogLog Unions

February, 26 2013

In the article from yesterday we talked about PostgreSQL HyperLogLog with some details. The real magic of that extension has been skimmed over though, and needs another very small article all by itself, in case you missed it.

 

PostgreSQL HyperLogLog

February, 25 2013

If you've been following along at home the newer statistics developments, you might have heard about this new State of The Art Cardinality Estimation Algorithm called HyperLogLog. This technique is now available for PostgreSQL in the extension postgresql-hll available at https://github.com/aggregateknowledge/postgresql-hll and soon to be in debian.

 

PostgreSQL for developers

November, 02 2012

As Guillaume says, we've been enjoying a great evening conference in Lyon 2 days ago, presenting PostgreSQL to developers. He did the first hour presenting the project and the main things you want to know to start using PostgreSQL in production, then I took the opportunity to be talking to developers to show off some SQL.

 

Reset Counter

October, 05 2012

I've been given a nice puzzle that I think is a good blog article opportunity, as it involves some thinking and window functions.

 

Let's say you need to ALTER TABLE foo ALTER COLUMN bar TYPE bigint;, and PostgreSQL is helpfully telling you that no you can't because such and such views depend on the column. The basic way to deal with that is to copy paste from the error message the names of the views involved, then prepare a script wherein you first DROP VIEW ...; then ALTER TABLE and finally CREATE VIEW again, all in the same transaction.

 

Dynamic Triggers in PLpgSQL

November, 24 2010

You certainly know that implementing dynamic triggers in PLpgSQL is impossible. But I had a very bad night, being up from as soon as 3:30 am today, so that when a developer asked me about reusing the same trigger function code from more than one table and for a dynamic column name, I didn't remember about it being impossible.

 

The drawback of hosting a static only website is, obviously, the lack of comments. What happens actually, though, is that I receive very few comments by direct mail. As I don't get another spam source to cleanup, I'm left unconvinced that's such a drawback. I still miss the low probability of seeing blog readers exchange directly, but I think a tapoueh.org mailing list would be my answer, here...

 

Window Functions example

September, 09 2010

So, when 8.4 came out there was all those comments about how getting window functions was an awesome addition. Now, it seems that a lot of people seeking for help in #postgresql just don't know what kind of problem this feature helps solving. I've already been using them in some cases here in this blog, for getting some nice overview about Partitioning: relation size per “group”.

 

Happy Numbers

August, 30 2010

After discovering the excellent Gwene service, which allows you to subscribe to newsgroups to read RSS content ( blogs, planets, commits, etc), I came to read this nice article about Happy Numbers. That's a little problem that fits well an interview style question, so I first solved it yesterday evening in Emacs Lisp as that's the language I use the most those days.

 

Playing with bit strings

August, 26 2010

The idea of the day ain't directly from me, I'm just helping with a very thin subpart of the problem. The problem, I can't say much about, let's just assume you want to reduce the storage of MD5 in your database, so you want to abuse bit strings. A solution to use them works fine, but the datatype is still missing some facilities, for example going from and to hexadecimal representation in text.

 

This time, we are trying to figure out where is the bulk of the data on disk. The trick is that we're using DDL partitioning, but we want a “nice” view of size per partition set. Meaning that if you have for example a parent table foo with partitions foo_201006 and foo_201007, you would want to see a single category foo containing the accumulated size of all the partitions underneath foo.

 

This time we're having a database where sequences were used, but not systematically as a default value of a given column. It's mainly an historic bad idea, but you know the usual excuse with bad ideas and bad code: the first 6 months it's experimental, after that it's historic.

 

The problem was raised this week on IRC and this time again I felt it would be a good occasion for a blog entry: how to load an XML file content into a single field?

par dim@tapoueh.org (Dimitri Fontaine) le lundi 25 mai 2015 à 15h17

vendredi 22 mai 2015

Actualités PostgreSQL.fr

Nouvelles versions mineures

Le PostgreSQL Global Development Group a publié un correctif qui concerne de nombreuses fonctionnalités et failles de sécurité. Toutes les versions de PostgreSQL sont concernées. Les nouvelles versions sont 9.4.2, 9.3.7, 9.2.11, 9.1.16 et 9.0.20. La mise à jour contient le correctif d'une possible corruption de données sur les versions 9.3 et 9.4 de PostgreSQL. Il est recommandé de procéder à la mise à jour dès que possible.

Correctif de la corruption de données

Les utilisateurs des versions 9.3 et 9.4 sont concernés par ce correctif. Les versions 9.2 ou précédentes ne sont pas affectées. La mise à jour corrige le problème du bouclage d'identifiant « multixact » ("multixact wraparound"), qui induit une corruption ou une perte de données. Les utilisateurs dont le taux de transactions est très élevé (1 million par heure) sur des bases ayant de nombreuses clés étrangères sont particulièrement vulnérables. Il est plus que recommandé de procéder aux mises à jour des installations dans les prochains jours.

Correctifs de sécurité

Cette mise à jour corrige trois failles de sécurité rapportées ces derniers mois. Aucune n'est particulièrement urgente. Toutefois, il est conseillé aux utilisateurs de vérifier si leurs installations sont concernées.

   CVE-2015-3165 Double libération de mémoire après timeout d'authentification.
   CVE-2015-3166 Erreurs imprévues de la bibliothèque standard
   CVE-2015-3167 pgcrypto contient plusieurs messages d'erreur lors de déchiffrement avec une mauvaise clé.

Il est également conseillé aux utilisateurs des authentifications par Kerberos, GSSAPI, ou SSPI de positionner include_realm à 1 dans le fichier pg_hba.conf, cette valeur devenant la valeur par défaut dans les futures versions.

Pour plus d'informations sur ces failles, on peut se référer aux pages de sécurité sur le site de PostgreSQL.

Autres correctifs et améliorations

Une nouvelle version, qui n'est pas celle par défaut, de l'extension citext corrige les fonctions regexp_matches() précédemment non-documentées, pour les aligner sur les versions texte usuelles de ces fonctions. La version corrigée utilisant un type de retour différent de l'ancienne, il est nécessaire de tester les applications avant d'exécuter la commande "ALTER EXTENSION citext UPDATE".

Plus de 50 autres correctifs sont fournis par ces mises à jour. La plupart concernent l'ensemble des versions supportées.

On peut citer :

 * la traduction des dates et estampilles temporelles infinies en "infinity" lors de conversion en json ;
 * la correction des fonctions json/jsonb populate_record() et to_record() ;
 * la correction de la vérification des contraintes d'exclusion différée ;
 * l'amélioration de la planification des requêtes touchant plusieurs schémas ;
 * la correction de trois problèmes de jointures ;
 * la garantie de verrouillage correct lors de l'utilisation de barrières de sécurité ("security barriers") dans les vues ;
 * la correction du verrou mort au démarrage lorsque le paramètre max_prepared_transactions est trop faible ;
 * le parcours récusif par fsync() du répertoire de données après un crash ;
 * la correction du lanceur d'autovacuum qui pouvait ne pas s'arrêter ;
 * la gestion de signaux inattendus dans la fonction LockBufferForCleanup() ;
 * la correction du crash de COPY IN vers une table possédant des contraintes de vérification ;
 * la suppression de l'attente de réplication synchrone sur les transactions en lecture seule ;
 * la correction de deux problèmes sur les index hash ;
 * la suppression des fuites mémoire lors de vacuum sur les index GIN ; 
 * la correction de deux problèmes sur les « background workers » ;
 * plusieurs corrections sur la réplication par décodage logique ;
 * plusieurs corrections sur pg_dump et pg_ugrade.

Cette version inclut une mise à jour vers la version 2015d de tzdata, avec les mise à jour pour l'Egypte, la Mongolie, la Palestine, et des modifications historiques pour le Canada et le Chili.

Fin de vie proche pour la branche 9.0

La version 9.0 arrive en fin de vie en septembre 2015. Il est probable que cette mise à jour soit une des dernières de cette branche. Les utilisateurs de PostgreSQL 9.0 peuvent commencer à planifier la mise à jour vers une version plus récente. Voir la page d'information sur les versions (http://www.postgresql.org/support/versioning ) pour de plus amples informations sur les dates de fin de vie.

Comme pour les autres versions mineures, il n'est pas nécessaire d'extraire et recharger les bases ou d'utiliser pg_upgrade pour appliquer cette mise à jour. Il suffit d'arrêter PostgreSQL, et de mettre à jour les binaires.

Les utilisateurs de l'extension citext auront une commande à jouer.

Les utilisateurs ayant négligé les mises à jour précédentes auront peut-être quelques actions post-mise à jour à effectuer.

Liens :

 * Téléchargement
 * Notes de version
 * Page sécurité

par SAS le vendredi 22 mai 2015 à 13h44

lundi 18 mai 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 17 mai 2015

Sur Vimeo se trouve un canal italophone dédié à PostgreSQL : https://vimeo.com/user10626375/videos

La CLEI (conférence latino-américaine d'informatique) se tiendra à Arequipa (Pérou) du 19 au 23 octobre 2015 : https://sites.google.com/a/spc.org.pe/clei2015/

Informatica 2016, qui aura lieu à Cuba, recherche du contenu touchant à PostgreSQL pour ses ateliers open-source : http://www.informaticahabana.cu/

PostgreSQL Conference Europe 2015 aura lieu du 27 au 30 octobre au Vienna Marriott Hotel à Vienne (Autriche). L'appel à conférenciers est lancé jusqu'au 7 août : http://2015.pgconf.eu/

Le SwissPUG est à présent ouvert aux nouveaux membres : http://www.swisspug.org/wiki/index.php/Special:Registration

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Robert Haas a poussé :

  • Fix DetermineSafeOldestOffset for the case where there are no mxacts. Commit b69bf30b9bfacafc733a9ba77c9587cf54d06c0c failed to take into account the possibility that there might be no multixacts in existence at all. Report by Thomas Munro; patch by me. http://git.postgresql.org/pg/commitdiff/312747c224be4c0a880b5172a010e980f8b654ca
  • Advance the stop point for multixact offset creation only at checkpoint. Commit b69bf30b9bfacafc733a9ba77c9587cf54d06c0c advanced the stop point at vacuum time, but this has subsequently been shown to be unsafe as a result of analysis by myself and Thomas Munro and testing by Thomas Munro. The crux of the problem is that the SLRU deletion logic may get confused about what to remove if, at exactly the right time during the checkpoint process, the head of the SLRU crosses what used to be the tail. This patch, by me, fixes the problem by advancing the stop point only following a checkpoint. This has the additional advantage of making the removal logic work during recovery more like the way it works during normal running, which is probably good. At least one of the calls to DetermineSafeOldestOffset which this patch removes was already dead, because MultiXactAdvanceOldest is called only during recovery and DetermineSafeOldestOffset was set up to do nothing during recovery. That, however, is inconsistent with the principle that recovery and normal running should work similarly, and was confusing to boot. Along the way, fix some comments that previous patches in this area neglected to update. It's not clear to me whether there's any concrete basis for the decision to use only half of the multixact ID space, but it's neither necessary nor sufficient to prevent multixact member wraparound, so the comments should not say otherwise. http://git.postgresql.org/pg/commitdiff/f6a6c46d7fd72878d37c75d4a3215d5a62128d0b
  • Even when autovacuum=off, force it for members as we do in other cases. Thomas Munro, with some adjustments by me. http://git.postgresql.org/pg/commitdiff/04e6d3b877e060d8445eb653b7ea26b1ee5cec6b
  • Increase threshold for multixact member emergency autovac to 50%. Analysis by Noah Misch shows that the 25% threshold set by commit 53bb309d2d5a9432d2602c93ed18e58bd2924e15 is lower than any other, similar autovac threshold. While we don't know exactly what value will be optimal for all users, it is better to err a little on the high side than on the low side. A higher value increases the risk that users might exhaust the available space and start seeing errors before autovacuum can clean things up sufficiently, but a user who hits that problem can compensate for it by reducing autovacuum_multixact_freeze_max_age to a value dependent on their average multixact size. On the flip side, if the emergency cap imposed by that patch kicks in too early, the user will experience excessive wraparound scanning and will be unable to mitigate that problem by configuration. The new value will hopefully reduce the risk of such bad experiences while still providing enough headroom to avoid multixact member exhaustion for most users. Along the way, adjust the documentation to reflect the effects of commit 04e6d3b877e060d8445eb653b7ea26b1ee5cec6b, which taught autovacuum to run for multixact wraparound even when autovacuum is configured off. http://git.postgresql.org/pg/commitdiff/b4d4ce1d50bbdf82cd2e2c1c7172b936df01c01d
  • Remove useless assertion. Here, snapshot->xcnt is an unsigned type, so it will always be non-negative. http://git.postgresql.org/pg/commitdiff/ae6157164faf5ec1636a9acfe18bfd28a31db098
  • Extend abbreviated key infrastructure to datum tuplesorts. Andrew Gierth, reviewed by Peter Geoghegan and by me. http://git.postgresql.org/pg/commitdiff/78efd5c1edb59017f06ef96773e64e6539bfbc86
  • Fix comment. Commit 78efd5c1edb59017f06ef96773e64e6539bfbc86 overlooked this. Report by Peter Geoghegan. http://git.postgresql.org/pg/commitdiff/61f68e0bed63aa5090e8be7c912843e49b30fc1e
  • doc: CREATE FOREIGN TABLE now allows CHECK ( ... ) NO INHERIT. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/92edba2665ae7bf43ed03538311e63652f9e2373

Bruce Momjian a poussé :

Tom Lane a poussé :

  • Fix incorrect checking of deferred exclusion constraint after a HOT update. If a row that potentially violates a deferred exclusion constraint is HOT-updated later in the same transaction, the exclusion constraint would be reported as violated when the check finally occurs, even if the row(s) the new row originally conflicted with have since been removed. This happened because the wrong TID was passed to check_exclusion_constraint(), causing the live HOT-updated row to be seen as a conflicting row rather than recognized as the row-under-test. Per bug #13148 from Evan Martin. It's been broken since exclusion constraints were invented, so back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/20781765f77c1fb6465aba97d211636ce92e7a0e
  • Add support for doing late row locking in FDWs. Previously, FDWs could only do "early row locking", that is lock a row as soon as it's fetched, even though local restriction/join conditions might discard the row later. This patch adds callbacks that allow FDWs to do late locking in the same way that it's done for regular tables. To make use of this feature, an FDW must support the "ctid" column as a unique row identifier. Currently, since ctid has to be of type TID, the feature is of limited use, though in principle it could be used by postgres_fdw. We may eventually allow FDWs to specify another data type for ctid, which would make it possible for more FDWs to use this feature. This commit does not modify postgres_fdw to use late locking. We've tested some prototype code for that, but it's not in committable shape, and besides it's quite unclear whether it actually makes sense to do late locking against a remote server. The extra round trips required are likely to outweigh any benefit from improved concurrency. Etsuro Fujita, reviewed by Ashutosh Bapat, and hacked up a lot by me http://git.postgresql.org/pg/commitdiff/afb9249d06f47d7a6d4a89fea0c3625fe43c5a5d
  • Fix postgres_fdw to return the right ctid value in EvalPlanQual cases. If a postgres_fdw foreign table is a non-locked source relation in an UPDATE, DELETE, or SELECT FOR UPDATE/SHARE, and the query selects its ctid column, the wrong value would be returned if an EvalPlanQual recheck occurred. This happened because the foreign table's result row was copied via the ROW_MARK_COPY code path, and EvalPlanQualFetchRowMarks just unconditionally set the reconstructed tuple's t_self to "invalid". To fix that, we can have EvalPlanQualFetchRowMarks copy the composite datum's t_ctid field, and be sure to initialize that along with t_self when postgres_fdw constructs a tuple to return. If we just did that much then EvalPlanQualFetchRowMarks would start returning "(0,0)" as ctid for all other ROW_MARK_COPY cases, which perhaps does not matter much, but then again maybe it might. The cause of that is that heap_form_tuple, which is the ultimate source of all composite datums, simply leaves t_ctid as zeroes in newly constructed tuples. That seems like a bad idea on general principles: a field that's really not been initialized shouldn't appear to have a valid value. So let's eat the trivial additional overhead of doing "ItemPointerSetInvalid(&(td->t_ctid))" in heap_form_tuple. This closes out our handling of Etsuro Fujita's report that tableoid and ctid weren't correctly set in postgres_fdw EvalPlanQual cases. Along the way we did a great deal of work to improve FDWs' ability to control row locking behavior; which was not wasted effort by any means, but it didn't end up being a fix for this problem because that feature would be too expensive for postgres_fdw to use all the time. Although the fix for the tableoid misbehavior was back-patched, I'm hesitant to do so here; it seems far less likely that people would care about remote ctid than tableoid, and even such a minor behavioral change as this in heap_form_tuple is perhaps best not back-patched. So commit to HEAD only, at least for the moment. Etsuro Fujita, with some adjustments by me http://git.postgresql.org/pg/commitdiff/0bb8528b5c738b45d0b65844750588c16bf75c52
  • Fix distclean/maintainer-clean targets to remove top-level tmp_install dir. The top-level makefile removes tmp_install in its "clean" target, but the distclean and maintainer-clean targets overlooked that (and they don't simply invoke clean, because that would result in an extra tree traversal). While at it, let's just make sure that removing GNUmakefile itself is the very last step of the recipe. http://git.postgresql.org/pg/commitdiff/9660710e2f5bbbf1b18640fbc5edcceafe7a10ad
  • Support "expanded" objects, particularly arrays, for better performance. This patch introduces the ability for complex datatypes to have an in-memory representation that is different from their on-disk format. On-disk formats are typically optimized for minimal size, and in any case they can't contain pointers, so they are often not well-suited for computation. Now a datatype can invent an "expanded" in-memory format that is better suited for its operations, and then pass that around among the C functions that operate on the datatype. There are also provisions (rudimentary as yet) to allow an expanded object to be modified in-place under suitable conditions, so that operations like assignment to an element of an array need not involve copying the entire array. The initial application for this feature is arrays, but it is not hard to foresee using it for other container types like JSON, XML and hstore. I have hopes that it will be useful to PostGIS as well. In this initial implementation, a few heuristics have been hard-wired into plpgsql to improve performance for arrays that are stored in plpgsql variables. We would like to generalize those hacks so that other datatypes can obtain similar improvements, but figuring out some appropriate APIs is left as a task for future work. (The heuristics themselves are probably not optimal yet, either, as they sometimes force expansion of arrays that would be better left alone.) Preliminary performance testing shows impressive speed gains for plpgsql functions that do element-by-element access or update of large arrays. There are other cases that get a little slower, as a result of added array format conversions; but we can hope to improve anything that's annoyingly bad. In any case most applications should see a net win. Tom Lane, reviewed by Andres Freund http://git.postgresql.org/pg/commitdiff/1dc5ebc9077ab742079ce5dac9a6664248d42916
  • Suppress uninitialized-variable warning. http://git.postgresql.org/pg/commitdiff/6c9e93d3ffbf99435636103ad69d6469c64e2aef
  • Docs: fix erroneous claim about max byte length of GB18030. This encoding has characters up to 4 bytes long, not 2. http://git.postgresql.org/pg/commitdiff/333d0779627a6c6125c018ea39da4427d3bdd93f
  • Fix portability issue in pg_audit. "%ld" is not a portable way to print int64's. This may explain the buildfarm crashes we're seeing --- it seems to make dromedary happy, at least. http://git.postgresql.org/pg/commitdiff/35a1e1d1593f4355c9d87bbc8208a8736801a607
  • Teach UtfToLocal/LocalToUtf to support algorithmic encoding conversions. Until now, these functions have only supported encoding conversions using lookup tables, which is fine as long as there's not too many code points to convert. However, GB18030 expects all 1.1 million Unicode code points to be convertible, which would require a ridiculously-sized lookup table. Fortunately, a large fraction of those conversions can be expressed through arithmetic, ie the conversions are one-to-one in certain defined ranges. To support that, provide a callback function that is used after consulting the lookup tables. (This patch doesn't actually change anything about the GB18030 conversion behavior, just provide infrastructure for fixing it.) Since this requires changing the APIs of UtfToLocal/LocalToUtf anyway, take the opportunity to rearrange their argument lists into what seems to me a saner order. And beautify the call sites by using lengthof() instead of error-prone sizeof() arithmetic. In passing, also mark all the lookup tables used by these calls "const". This moves an impressive amount of stuff into the text segment, at least on my machine, and is safer anyhow. http://git.postgresql.org/pg/commitdiff/7730f48ede0d222e7f750541d3d5f0f74d75d99b
  • Honor traditional SGML NAMELEN limit. We've conformed to this limit in the past, so might as well continue to. Aaron Swenson http://git.postgresql.org/pg/commitdiff/4b8f797f672bef07b4e87b4650b4035731b61d84
  • Fix insufficiently-paranoid GB18030 encoding verifier. The previous coding effectively only verified that the second byte of a multibyte character was in the expected range; moreover, it wasn't careful to make sure that the second byte even exists in the buffer before touching it. The latter seems unlikely to cause any real problems in the field (in particular, it could never be a problem with null-terminated input), but it's still a bug. Since GB18030 is not a supported backend encoding, the only thing we'd really be doing with GB18030 text is converting it to UTF8 in LocalToUtf, which would fail anyway on any invalid character for lack of a match in its lookup table. So the only user-visible consequence of this change should be that you'll get "invalid byte sequence for encoding" rather than "character has no equivalent" for malformed GB18030 input. However, impending changes to the GB18030 conversion code will require these tighter up-front checks to avoid producing bogus results. http://git.postgresql.org/pg/commitdiff/a868931fecdf93f3ceb1c9431bb93757b706269d
  • Fix outdated src/test/mb/ tests, and add a GB18030 test. The expected-output files for these tests were broken by the recent addition of a warning for hash indexes. Update them. Also add a test case for GB18030 encoding, similar to the other ones. This is a pretty weak test, but it's better than nothing. http://git.postgresql.org/pg/commitdiff/07af523870bcfe930134054febd3a6a114942e5b
  • Extend GB18030 encoding conversion to cover full Unicode range. Our previous code for GB18030 <-> UTF8 conversion only covered Unicode code points up to U+FFFF, but the actual spec defines conversions for all code points up to U+10FFFF. That would be rather impractical as a lookup table, but fortunately there is a simple algorithmic conversion between the additional code points and the equivalent GB18030 byte patterns. Make use of the just-added callback facility in LocalToUtf/UtfToLocal to perform the additional conversions. Having created the infrastructure to do that, we can use the same code to map certain linearly-related subranges of the Unicode space below U+FFFF, allowing removal of the corresponding lookup table entries. This more than halves the lookup table size, which is a substantial savings; utf8_and_gb18030.so drops from nearly a megabyte to about half that. In support of doing that, replace ISO10646-GB18030.TXT with the data file gb-18030-2000.xml (retrieved from http://source.icu-project.org/repos/icu/data/trunk/charset/data/xml/ ) in which these subranges have been deleted from the simple lookup entries. Per bug #12845 from Arjen Nienhuis. The conversion code added here is based on his proposed patch, though I whacked it around rather heavily. http://git.postgresql.org/pg/commitdiff/8d3e0906df5496b853cc763f87b9ffd2ae27adbe
  • Fix uninitialized variable. Per compiler warnings. http://git.postgresql.org/pg/commitdiff/66493dd7aa02e19e93f7d5687acaab70075db34f
  • Improve test for CONVERT() with GB18030 <-> UTF8. Add a bit of coverage of high code points. Arjen Nienhuis http://git.postgresql.org/pg/commitdiff/199f5973c50fe94e128508ff2218c42126fd0ee1
  • Update time zone data files to tzdata release 2015d. DST law changes in Egypt, Mongolia, Palestine. Historical corrections for Canada and Chile. Revised zone abbreviation for America/Adak (HST/HDT not HAST/HADT). http://git.postgresql.org/pg/commitdiff/9d366c1f3d758c3b80cd482a6d4528960c4fc325
  • Avoid direct use of INFINITY. It's not very portable. Per buildfarm. http://git.postgresql.org/pg/commitdiff/12cc299c65ba6124c7d459f2605404ad43909db3
  • More portability fixing for bipartite_match.c. <float.h> is required for isinf() on some platforms. Per buildfarm. http://git.postgresql.org/pg/commitdiff/26058bf0dc5fca0a5107b2ee136a81a86cf36049
  • Fix docs typo. I don't think "respectfully" is what was meant here... http://git.postgresql.org/pg/commitdiff/c65aa7a87e4232d7eefea3d78ec7e1d9f8b4708b
  • First-draft release notes for 9.4.2 et al. As usual, the release notes for older branches will be made by cutting these down, but put them up for community review first. http://git.postgresql.org/pg/commitdiff/0563b4c0c3b7cc2323cfb63e11d723764e2d5f7d
  • Release notes for 9.4.2, 9.3.7, 9.2.11, 9.1.16, 9.0.20. http://git.postgresql.org/pg/commitdiff/a0891d2d0136ea06cde957635338c0c238df87de

Stephen Frost a poussé :

Peter Eisentraut a poussé :

Álvaro Herrera a poussé :

  • "Fix" test_ddl_deparse regress test schedule. MSVC is not smart enough to figure it out, so dumb down the Makefile and remove the schedule file. Also add a .gitignore file. Author: Michael Paquier http://git.postgresql.org/pg/commitdiff/007c932e5aaf6d68087f134b8557bbb7db149e94
  • Allow on-the-fly capture of DDL event details. This feature lets user code inspect and take action on DDL events. Whenever a ddl_command_end event trigger is installed, DDL actions executed are saved to a list which can be inspected during execution of a function attached to ddl_command_end. The set-returning function pg_event_trigger_ddl_commands can be used to list actions so captured; it returns data about the type of command executed, as well as the affected object. This is sufficient for many uses of this feature. For the cases where it is not, we also provide a "command" column of a new pseudo-type pg_ddl_command, which is a pointer to a C structure that can be accessed by C code. The struct contains all the info necessary to completely inspect and even reconstruct the executed command. There is no actual deparse code here; that's expected to come later. What we have is enough infrastructure that the deparsing can be done in an external extension. The intention is that we will add some deparsing code in a later release, as an in-core extension. A new test module is included. It's probably insufficient as is, but it should be sufficient as a starting point for a more complete and future-proof approach. Authors: Álvaro Herrera, with some help from Andres Freund, Ian Barwick, Abhijit Menon-Sen. Reviews by Andres Freund, Robert Haas, Amit Kapila, Michael Paquier, Craig Ringer, David Steele. Additional input from Chris Browne, Dimitri Fontaine, Stephen Frost, Petr Jelínek, Tom Lane, Jim Nasby, Steven Singer, Pavel Stěhule. Based on original work by Dimitri Fontaine, though I didn't use his code. Discussion: https://www.postgresql.org/message-id/m2txrsdzxa.fsf@2ndQuadrant.fr https://www.postgresql.org/message-id/20131108153322.GU5809@eldon.alvh.no-ip.org https://www.postgresql.org/message-id/20150215044814.GL3391@alvh.no-ip.org http://git.postgresql.org/pg/commitdiff/b488c580aef4e05f39be5daaab6464da5b22a494
  • Move strategy numbers to include/access/stratnum.h. For upcoming BRIN opclasses, it's convenient to have strategy numbers defined in a single place. Since there's nothing appropriate, create it. The StrategyNumber typedef now lives there, as well as existing strategy numbers for B-trees (from skey.h) and R-tree-and-friends (from gist.h). skey.h is forced to include stratnum.h because of the StrategyNumber typedef, but gist.h is not; extensions that currently rely on gist.h for rtree strategy numbers might need to add a new A few .c files can stop including skey.h and/or gist.h, which is a nice side benefit. Per discussion: https://www.postgresql.org/message-id/20150514232132.GZ2523@alvh.no-ip.org Authored by Emre Hasegeli and Álvaro. (It's not clear to me why bootscanner.l has any #include lines at all.) http://git.postgresql.org/pg/commitdiff/26df7066cc229887d4defdf1d105c0a22b8a88fb
  • Add BRIN infrastructure for "inclusion" opclasses. This lets BRIN be used with R-Tree-like indexing strategies. Also provided are operator classes for range types, box and inet/cidr. The infrastructure provided here should be sufficient to create operator classes for similar datatypes; for instance, opclasses for PostGIS geometries should be doable, though we didn't try to implement one. (A box/point opclass was also submitted, but we ripped it out before commit because the handling of floating point comparisons in existing code is inconsistent and would generate corrupt indexes.) Author: Emre Hasegeli. Cosmetic changes by me. Review: Andreas Karlsson http://git.postgresql.org/pg/commitdiff/b0b7be61337fc64147f2ad0af5bf2c0e6b8a709f

Andrew Dunstan a poussé :

  • pg_basebackup -F t now succeeds with a long symlink target http://git.postgresql.org/pg/commitdiff/97e0aa697983cf7f7f79e69f2dc248fdefb7dbf6
  • Map basebackup tablespaces using a tablespace_map file. Windows can't reliably restore symbolic links from a tar format, so instead during backup start we create a tablespace_map file, which is used by the restoring postgres to create the correct links in pg_tblspc. The backup protocol also now has an option to request this file to be included in the backup stream, and this is used by pg_basebackup when operating in tar mode. This is done on all platforms, not just Windows. This means that pg_basebackup will not not work in tar mode against 9.4 and older servers, as this protocol option isn't implemented there. Amit Kapila, reviewed by Dilip Kumar, with a little editing from me. http://git.postgresql.org/pg/commitdiff/72d422a5227ef6f76f412486a395aba9f53bf3f0
  • Fix some errors from jsonb functions patch. The catalog version should have been bumped, and the alternative regression result file was not up to date with the name of jsonb_pretty. http://git.postgresql.org/pg/commitdiff/5c7df74204e2fb9440b576518d40fcf3ac65c8ac
  • Fix jsonb replace and delete on scalars and empty structures. These operations now error out if attempted on scalars, and simply return the input if attempted on empty arrays or objects. Along the way we remove the unnecessary cloning of the input when it's known to be unchanged. Regression tests covering these cases are added. http://git.postgresql.org/pg/commitdiff/3f2cec797ecceb7467e365410506c0817f9d0163
  • Additional functions and operators for jsonb. jsonb_pretty(jsonb) produces nicely indented json output. jsonb || jsonb concatenates two jsonb values. jsonb - text removes a key and its associated value from the json jsonb - int removes the designated array element jsonb - text[] removes a key and associated value or array element at the designated path jsonb_replace(jsonb,text[],jsonb) replaces the array element designated by the path or the value associated with the key designated by the path with the given value. Original work by Dmitry Dolgov, adapted and reworked for PostgreSQL core by Andrew Dunstan, reviewed and tidied up by Petr Jelinek. http://git.postgresql.org/pg/commitdiff/c6947010ceb42143d9f047c65c1eac2b38928ab7

Andres Freund a poussé :

  • Fix ON CONFLICT bugs that manifest when used in rules. Specifically the tlist and rti of the pseudo "excluded" relation weren't properly treated by expression_tree_walker, which lead to errors when excluded was referenced inside a rule because the varnos where not properly adjusted. Similar omissions in OffsetVarNodes and expression_tree_mutator had less impact, but should obviously be fixed nonetheless. A couple tests of for ON CONFLICT UPDATE into INSERT rule bearing relations have been added. In passing I updated a couple comments. http://git.postgresql.org/pg/commitdiff/4af6e61a363246cf7fff3368a76603b0ce9945dd
  • Add pgstattuple_approx() to the pgstattuple extension. The new function allows to estimate bloat and other table level statics in a faster, but approximate, way. It does so by using information from the free space map for pages marked as all visible in the visibility map. The rest of the table is actually read and free space/bloat is measured accurately. In many cases that allows to get bloat information much quicker, causing less IO. Author: Abhijit Menon-Sen Reviewed-By: Andres Freund, Amit Kapila and Tomas Vondra Discussion: 20140402214144.GA28681@kea.toroid.org http://git.postgresql.org/pg/commitdiff/5850b20f58a594ac69f4f77b24cad94fc3bfd946
  • Support GROUPING SETS, CUBE and ROLLUP. This SQL standard functionality allows to aggregate data by different GROUP BY clauses at once. Each grouping set returns rows with columns grouped by in other sets set to NULL. This could previously be achieved by doing each grouping as a separate query, conjoined by UNION ALLs. Besides being considerably more concise, grouping sets will in many cases be faster, requiring only one scan over the underlying data. The current implementation of grouping sets only supports using sorting for input. Individual sets that share a sort order are computed in one pass. If there are sets that don't share a sort order, additional sort & aggregation steps are performed. These additional passes are sourced by the previous sort step; thus avoiding repeated scans of the source data. The code is structured in a way that adding support for purely using hash aggregation or a mix of hashing and sorting is possible. Sorting was chosen to be supported first, as it is the most generic method of implementation. Instead of, as in an earlier versions of the patch, representing the chain of sort and aggregation steps as full blown planner and executor nodes, all but the first sort are performed inside the aggregation node itself. This avoids the need to do some unusual gymnastics to handle having to return aggregated and non-aggregated tuples from underlying nodes, as well as having to shut down underlying nodes early to limit memory usage. The optimizer still builds Sort/Agg node to describe each phase, but they're not part of the plan tree, but instead additional data for the aggregation node. They're a convenient and preexisting way to describe aggregation and sorting. The first (and possibly only) sort step is still performed as a separate execution step. That retains similarity with existing group by plans, makes rescans fairly simple, avoids very deep plans (leading to slow explains) and easily allows to avoid the sorting step if the underlying data is sorted by other means. A somewhat ugly side of this patch is having to deal with a grammar ambiguity between the new CUBE keyword and the cube extension/functions named cube (and rollup). To avoid breaking existing deployments of the cube extension it has not been renamed, neither has cube been made a reserved keyword. Instead precedence hacking is used to make GROUP BY cube(..) refer to the CUBE grouping sets feature, and not the function cube(). To actually group by a function cube(), unlikely as that might be, the function name has to be quoted. Needs a catversion bump because stored rules may change. Author: Andrew Gierth and Atri Sharma, with contributions from Andres Freund Reviewed-By: Andres Freund, Noah Misch, Tom Lane, Svenne Krap, Tomas Vondra, Erik Rijkers, Marti Raudsepp, Pavel Stehule Discussion: CAOeZVidmVRe2jU6aMk_5qkxnB7dfmPROzM7Ur8JPW5j8Y5X-Lw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/f3d3118532175541a9a96ed78881a3b04a057128

Fujii Masao a poussé :

Simon Riggs a poussé :

Heikki Linnakangas a poussé :

  • Allow GiST distance function to return merely a lower-bound. The distance function can now set *recheck = false, like index quals. The executor will then re-check the ORDER BY expressions, and use a queue to reorder the results on the fly. This makes it possible to do kNN-searches on polygons and circles, which don't store the exact
  • Fix datatype confusion with the new lossy GiST distance functions. We can only support a lossy distance function when the distance function's datatype is comparable with the original ordering operator's datatype. The distance function always returns a float8, so we are limited to float8, and float4 (by a hard-coded cast of the float8 to float4). In light of this limitation, it seems like a good idea to have a separate 'recheck' flag for the ORDER BY expressions, so that if you have a non-lossy distance function, it still works with lossy quals. There are cases like that with the build-in or contrib opclasses, but it's plausible. There was a hidden assumption that the ORDER BY values returned by GiST match the original ordering operator's return type, but there are plenty of examples where that's not true, e.g. in btree_gist and pg_trgm. As long as the distance function is not lossy, we can tolerate that and just not return the distance to the executor (or rather, always return NULL). The executor doesn't need the distances if there are no lossy results. There was another little bug: the recheck variable was not initialized before calling the distance function. That revealed the bigger issue, as the executor tried to reorder tuples that didn't need reordering, and that failed because of the datatype mismatch. http://git.postgresql.org/pg/commitdiff/98edd617f3b62a02cb2df9b418fcc4ece45c7ec0
  • Fix docs build. Oops. http://git.postgresql.org/pg/commitdiff/8b0f105d2d179dc1085b16f0594c8fa78d13267d
  • Add archive_mode='always' option. In 'always' mode, the standby independently archives all files it receives from the primary. Original patch by Fujii Masao, docs and review by me. http://git.postgresql.org/pg/commitdiff/ffd37740ee6fcd434416ec0c5461f7040e0a11de
  • Silence another create_index regression test failure. More platform differences in the less-significant digits in output. Per buildfarm member rover_firefly, still. http://git.postgresql.org/pg/commitdiff/11a83bbedd73800db70f6f2af5a8eb10d15d39d7
  • Silence create_index regression test failure. The expected output contained some floating point values which might get rounded slightly differently on different platforms. The exact output isn't very interesting in this test, so just round it. Per buildfarm member rover_firefly. http://git.postgresql.org/pg/commitdiff/9feaba28e27820f91d13c3de6581bb3b8c3234c6
  • value in the index, but just a bounding box. Alexander Korotkov and me http://git.postgresql.org/pg/commitdiff/35fcb1b3d038a501f3f4c87c05630095abaaadab

Magnus Hagander a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Andres Freund sent in another revision of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Michael Paquier sent in a patch to improve regression tests to check LOCK TABLE and table permissions.
  • Michael Paquier sent in a patch to remove leftovers after dcae5fac (improve speed of make check-world) in git tree with check-world.
  • Abhijit Menon-Sen sent in two more revisions of a patch to create a fast bloat measurement tool.
  • Heikki Linnakangas sent in two more revisions of a patch to fix some interactions between streaming replication and WAL archiving.
  • Kaigai Kouhei sent in three more revisions of a patch to improve the custom/foreign join API.
  • Stephen Frost sent in two more revisions of a patch to add default roles for functionality.
  • Álvaro Herrera and Petr Jelinek traded patches to add an access method for sequences.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add multi-column statistics.
  • Petr A. Korobeinikov sent in another revision of a patch to support for \ev viewname and \sv viewname in psql.
  • Fabien COELHO sent in a patch to allow backslash-continuations in custom scripts.
  • Etsuro Fujita sent in a doc patch for ALTER FOREIGN TABLE.
  • Robert Haas sent in a patch to fix the documentation of BackgroundWorkerInitializeConnection.
  • Beena Emerson sent in another revision of a patch to add support for N synchronous standby servers.
  • Shigeru HANADA sent in another revision of a patch to add postgres_fdw join pushdown.

par N Bougain le lundi 18 mai 2015 à 21h40

Adrien Nayrat

Partition xlogs pleine

Cette semaine j’ai été confronté à un incident sur une instance postgres. La partition xlogs était pleine :

PANIC: could not write to file "pg_xlog/xlogtemp.12352": No space left on device
LOG: startup process (PID 12352) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure

Cette instance est répliquée sur un autre serveur et archive ses journaux de transaction sur un troisième serveur. Ce dernier était plein, Postgres est intelligent et a conservé ses journaux de transaction en attendant de pouvoir les archiver… Jusqu’à ce que la partition des journaux se retrouve pleine entraînant l’arrêt de Postgres.

Le nettoyage sur le serveur d’archivage n’était pas suffisant, il fallait libérer de la place sur la partition. La solution qui vient en tête est d’agrandir la partition. Je ne souhaitais pas agrandir la partition. J’ai donc fait un truc tout simple : Déplacer un fichier journal et faire un lien symbolique vers celui-ci. Postgres est reparti et j’ai lancé un “CHECKPOINT” immédiatement ce qui a libéré de la place. Ensuite il faut penser à supprimer le fichier car Postgres a juste supprimé le lien.
C’est plus rapide à mettre en place que d’agrandir la partition.

En fouillant sur google je suis tombé sur ce site :
Solving pg_xlog out of disk space problem on Postgres

L’auteur nous donne une astuce toute simple : Créer un fichier plein de vide. Si un jour la partition est pleine il suffit de le supprimer pour faire repartir l’instance :

 dd if=/dev/zero of=DO_NOT_MOVE_THIS_FILE bs=1MB count=100

Bilan : Il faut bien penser à superviser l’archivage de Postgres!

FacebookTwitterGoogle+ViadeoPrintEmailGoogle GmailLinkedInPartager

par Adrien Nayrat le lundi 18 mai 2015 à 20h00

dimanche 17 mai 2015

ulhume

Rooter le LG G Pad 7.0

Lorsque l&aposon achète un device Android, se pose toujours le même problème &aposcomment vais-je rooter ce machin là&apos.

Voici donc comment faire pour la tablette LG GPad 7.0.

Pourquoi faire ?

Comme mon sujet pour l&aposannée 2015 est clairement le développement d&aposapplications mobile (allez zouh, un peu de pub ;-), après m&aposêtre commis à acheter un Mac Mini et un iPad Mini, j&aposai aussi dut faire l&aposacquisition d&aposun fairphone (pour me racheter un peu...) et d&aposune tablette Android, le LG G Pad 7.0 (le Windows Phone... m&aposa été gracieusement donné, ouf !! ;-).

Concernant les deux devices Android, seul le fairphone a le bon goût d&aposêtre rooté en sortie d&aposusine. En revanche, le LG, c&aposest prison dorée... Or pour développer sous Android, rooter est l&aposétape nécessaire. Car outre l&aposaspect philosophique (je ne supporte pas l&aposidée qu&aposon m&aposempêche de faire ce que je veux de ce que j&aposachète), l&aposaccès root me permet d&aposutiliser ADB en mode réseau (c&aposest à dire sans cable USB). Et ça, je ne sais pas m&aposen passer...

Bref, trêve de blabla, voyons comment casser la bête...

Passer en mode développeur

La première étape consiste déjà à activer le mode développeur, ce qui n&aposest pas aussi évident que l&aposon pourrait se l&aposimaginer. Pour cela vous devrez aller dans les réglages, puis dans le menu à propos de la tablette, et enfin dans information sur le logiciel. Arrivé là, accrochez-vous bien, vous devez taper plusieurs fois sur le Numéro de build.

La tablette vous demande alors si vous êtes certain de votre action (non, non, j&aposai tapé 20 fois là dessus par pur hasard...), puis fera apparaître, à la positive, le menu tant utile pour les développeurs.

Dans ce menu, vous allez devoir vous rendre dans la section debuggage pour activer le debuggage de la connection USB. Ce qui est le pré-requis indispensable pour pouvoir utiliser ADB mais aussi pour pouvoir rooter l&aposappareil.

Cas n°1, KitKat

Récupération de Purple Drake

Pour ceux qui ont acquis leur tablette à sa sortie, la version Android de base est KitKat. Pour rooter cette version, le sésame s&aposappelle Purple Drake. Purple Drake Pour le rooting à proprement parlé, le sésame s&aposappelle Purple Drake. Téléchargez donc la dernière version (R3 dans mon cas) et décompressez là en local.

Pour ceux qui utilisent (encore) Wheezy

Pour ceux qui sont sur une version récente du kernel, cette étape peut être zappée. Dans mon cas, la version de GLibC incluse dans Debian Wheezy ne me permet de lancer le script tel quel avec les versions binaires d&aposadb incluse dans l&aposarchive.

Cela se règle cependant simplement en installant la version debian d&aposadb par sudo apt-get install android-adb. Ceci fait, allez dans le dossier assets de l&aposarchive décompressée de Purple Drake et éditez le fichier purpledrake_main.sh pour remplacer en ligne 16, $1 par /usr/bin.

Lorsque tout est en ordre, il ne reste plus qu&aposà lancer l&aposoutil. Assurez-vous que la tablette n&aposa aucune application de lancée (rebootez là si nécessaire), que le cable USB est bien connecté, et que dans la barre de status de la tablette vous voyez bien que la connection USB se fait en mode debuggage.

root !

Si tout est OK, lancez le rooting par sudo ./purpledrake_linux.sh. La raison du sudo ici est que sous Debian, en standard, seul root a accès au device USB. Cela peut se configurer au niveau d&aposudev mais ça me saoule un peu d&aposavoir à faire cela à chaque fois, sur chaque machine et pour chaque device. Et c&aposest d&aposailleurs l&aposune des raisons qui me fait passer ADB en mode réseau.

Une fois l&aposoutil lancé il suffit de se laisser guider par le script. Il va d&aposabord rebooter le device, puis installer un root temporaire, puis l&aposutiliser, si vous le désirez, pour mettre en place le root permanent.

Lorsque la tablette a redémarré, tout application qui demande l&aposaccès root devrait ainsi l&aposobtenir. C&aposest une première étape mais ce n&aposest pas très sécurisé, loin de là.

Super pouvoirs à la demande

Pour aller un cran plus loin, rendez vous sur le market et téléchargez SuperSU. Cette application au démarrage va remplacer la commande su fournie par Purple Drake, par une version qui vous demandera si telle application a bien le droit d&aposobtenir l&aposaccès root.

Au premier lancement, SuperSU va vous proposer une procédure de remplacement de la commande en mode normal ou recovery. J&aposai personnellement pris l&aposoption normal.

Et après une installation avec succès suivi d&aposun redémarrage de l&aposengin, tout était opérationnel.

Félicitation, votre device vous appartient !

Cas n°2, Lollipop

Si vous avez acheté votre LG plus récemment, ou si, comme moi, vous avez clické sur "mettre à jour" sans faire tourner votre cerveau, vous n&aposêtes pas sous Kitkat mais sous Lollipop. Et là, le purple drake il marche plus du tout !

Alors heureusement il y a une autre méthode qui fonctionne parfaitement mais elle nécessite... windows. Je sais, c&aposest pas cool, mais je n&aposai pas trouvé mieux.

Pour rooter, petite liste de courses :

  1. Les pilotes Android de chez LG. Perso, je les ai trouvés .
  2. Une petite application toute mignonette qui va vous simplifier le rootage, et que vous trouverez ici

Première étape, installer les pilotes sur le windows. Je ne sais pas pour vous mais chez moi cela a pris des plombes !!!

Ceci fait, connectez votre LG par le câble USB et lancez l&aposapplication puis clickez sur ROOT DEVICE. Cela va lancer une console et le script associé va redémarrer le device. Si après ce redémarrage vous vous retrouvez à nouveau sous votre home Android, c&aposest que quelque chose n&aposaura pas marché. Dans ce cas le script va (sans doute) vous dire (je traduis ;-) "désolé, mais j&aposai échoué salement à passer le device en mode série, fait le toi-même, moi j&aposattends".

En effet, le script a besoin que le device soit dans un mode très spécial de mise à jour du firmeware qui place le port USB en mode "port série". Dans ce mode, il lui sera possible de lire un numéro de série qui permettra par la suite de rooter l&aposengin.

Pour faire cela manuellement procédez comme suit sans toucher au script côté windows qui doit afficher un waiting device. Qu&aposil attende donc...

  1. Débranchez le cable,
  2. Éteignez le device par une longue pression sur le bouton d&aposallumage, puis Éteindre,
  3. Pressez le bouton qui monte le volume (c&aposest celui qui est prêt du bouton d&aposallumage),
  4. Tout en maintenant ce bouton pressé, rebranchez le cable USB
  5. Là le device doit s&aposallumer et après un temps il va se stabiliser dans un mode étrange avec une sorte de barre de progression pour la mise à jour du firmware.
  6. Le script côté windows doit logiquement se réveiller et recommencer à vous causer,
  7. Lorsque le script a terminé son boulot, le device a redémarré en mode Android classique.
  8. Le script vous demande (en anglais) de l&aposarrêter (???) en pressant les touches Control+C , puis de pressez N puis Enter.
  9. Ceci fait, vous êtes de retours sur l&aposinterface graphique.

À ce stage, normalement c&aposest tout bon et le LG est rooté. Pour tester, déverrouillez la home et sur l&aposinterface côté Windows pressez le bouton "ADB SHELL". Vous devriez voir apparaître une console avec quelque chose comme &aposshell@e7wifi:/ $&apos. Maintenant tapez su puis validez. Et là, côté Android, une boite de dialogue doit apparaître pour vous demander si vous authoriser ce passage en mode root. Acceptez. De retour sous Windows, dans la console, devrait alors s&aposafficher root@e7wifi:/ #.

C&aposest bon, vous êtes root !

Tester le tout

Pour tester, le plus simple est d&aposinstaller ADB Wifi à partir du market et de le lancer. Si les étapes précédentes ont fonctionnées, lorsque vous activerez la ADB en mode réseau, une boite de dialogue doit apparaître pour valider l&aposaccès.

Ceci fait, ADB Wifi doit vous indiquer qu&aposil est en écoute et vous fournis l&aposIP de connection. Vous n&aposavez alors plus qu&aposà vous connecter de votre machine comme ceci :

$sudo adb connect 192.168.154.21
connected to 192.168.154.21:5555
$sudo adb shell

Conclusion

Il est toujours navrant d&aposavoir à perdre du temps sur quelque chose d&aposaussi trivial.

Espérons que peu à peu les constructeurs s&aposinspire de FairPhone et arrêtent ainsi de prendre leurs clients pour des crétins... On peut toujours rêver ;-)

Please enable JavaScript to view the comments powered by Disqus.

dimanche 17 mai 2015 à 22h13

lundi 11 mai 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 10 mai 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Second try at fixing warnings caused by commit 9b43d73b3f9bef27. Commit ef3f9e642d2b2bba suppressed one cause of warnings here, but recent clang on OS X is still unhappy because we're passing a "long" to abs(). The fact that tm_gmtoff is declared as long is no doubt a hangover from days when int might be only 16 bits; but Postgres has never been able to run on such machines, so we can just cast it to int with no worries. For consistency, also cast to int in the other uses of tm_gmtoff in this stanza. Note: this code is still broken on machines that don't follow C99 integer-division-truncates-towards-zero rules. Given the lack of complaints about it, I don't feel a large desire to complicate things enough to cope with the pre-C99 rules. http://git.postgresql.org/pg/commitdiff/c90b85e4d9e4ae3bc26459cc54697e1adaa4315f
  • Improve procost estimates for some text search functions. The text search functions that involve parsing raw text into lexemes are remarkably CPU-intensive, so estimating them at the same cost as most other built-in functions seems like a mistake; moreover, doing so turns out to discourage the optimizer from using functional indexes on these functions. After some debate, we've agreed to raise procost from 1 to 100 for to_tsvector(), plainto_tsvector(), to_tsquery(), ts_headline(), ts_match_tt(), and ts_match_tq(), which are all the text search functions that parse raw text. Also increase procost for the 2-argument form of ts_rewrite() (tsquery_rewrite_query); while this function doesn't do text parsing, it does execute a user-supplied SQL query, so its previous procost of 1 is clearly a drastic underestimate. It seems reasonable to assign it the same cost we assign to PL functions by default, so 100 is the number here too. I did not bother bumping catversion for this change, since it does not break catalog compatibility with the server executable nor result in any regression test changes. Per complaint from Andrew Gierth and subsequent discussion. http://git.postgresql.org/pg/commitdiff/2503982be4ca48f48d2bb6e1d46160b23e4bb268
  • Fix incorrect declaration of citext's regexp_matches() functions. These functions should return SETOF TEXT[], like the core functions they are wrappers for; but they were incorrectly declared as returning just TEXT[]. This mistake had two results: first, if there was no match you got a scalar null result, whereas what you should get is an empty set (zero rows). Second, the 'g' flag was effectively ignored, since you would get only one result array even if there were multiple matches, as reported by Jeff Certain. While ignoring 'g' is a clear bug, the behavior for no matches might well have been thought to be the intended behavior by people who hadn't compared it carefully to the core regexp_matches() functions. So we should tread carefully about introducing this change in the back branches. Still, it clearly is a bug and so providing some fix is desirable. After discussion, the conclusion was to introduce the change in a 1.1 version of the citext extension (as we would need to do anyway); 1.0 still contains the incorrect behavior. 1.1 is the default and only available version in HEAD, but it is optional in the back branches, where 1.0 remains the default version. People wishing to adopt the fix in back branches will need to explicitly do ALTER EXTENSION citext UPDATE TO '1.1'. (I also provided a downgrade script in the back branches, so people could go back to 1.0 if necessary.) This should be called out as an incompatible change in the 9.5 release notes, although we'll also document it in the next set of back-branch release notes. The notes should mention that any views or rules that use citext's regexp_matches() functions will need to be dropped before upgrading to 1.1, and then recreated again afterwards. Back-patch to 9.1. The bug goes all the way back to citext's introduction in 8.4, but pre-9.1 there is no extension mechanism with which to manage the change. Given the lack of previous complaints it seems unnecessary to change this behavior in 9.0, anyway. http://git.postgresql.org/pg/commitdiff/b22527f29dba6395a9e950fc655d34914c960f89
  • citext's regexp_matches() functions weren't documented, either. http://git.postgresql.org/pg/commitdiff/929ca96584bef1cc7d09a8e57d26d8c3f25a92a4
  • Add missing "static" marker. Per buildfarm member pademelon. http://git.postgresql.org/pg/commitdiff/c594c750789fd98a19dcdf974b87ba9833995cf5
  • Code review for foreign/custom join pushdown patch. Commit e7cb7ee14555cc9c5773e2c102efd6371f6f2005 included some design decisions that seem pretty questionable to me, and there was quite a lot of stuff not to like about the documentation and comments. Clean up as follows: * Consider foreign joins only between foreign tables on the same server, rather than between any two foreign tables with the same underlying FDW handler function. In most if not all cases, the FDW would simply have had to apply the same-server restriction itself (far more expensively, both for lack of caching and because it would be repeated for each combination of input sub-joins), or else risk nasty bugs. Anyone who's really intent on doing something outside this restriction can always use the set_join_pathlist_hook. * Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist to better reflect what they're for, and allow these custom scan tlists to be used even for base relations. * Change make_foreignscan() API to include passing the fdw_scan_tlist value, since the FDW is required to set that. Backwards compatibility doesn't seem like an adequate reason to expect FDWs to set it in some ad-hoc extra step, and anyway existing FDWs can just pass NIL. * Change the API of path-generating subroutines of add_paths_to_joinrel, and in particular that of GetForeignJoinPaths and set_join_pathlist_hook, so that various less-used parameters are passed in a struct rather than as separate parameter-list entries. The objective here is to reduce the probability that future additions to those parameter lists will result in source-level API breaks for users of these hooks. It's possible that this is even a small win for the core code, since most CPU architectures can't pass more than half a dozen parameters efficiently anyway. I kept root, joinrel, outerrel, innerrel, and jointype as separate parameters to reduce code churn in joinpath.c --- in particular, putting jointype into the struct would have been problematic because of the subroutines' habit of changing their local copies of that variable. * Avoid ad-hocery in ExecAssignScanProjectionInfo. It was probably all right for it to know about IndexOnlyScan, but if the list is to grow we should refactor the knowledge out to the callers. * Restore nodeForeignscan.c's previous use of the relcache to avoid extra GetFdwRoutine lookups for base-relation scans. * Lots of cleanup of documentation and missed comments. Re-order some code additions into more logical places. http://git.postgresql.org/pg/commitdiff/1a8a4e5cde2b7755e11bde2ea7897bd650622d3e

Andrew Dunstan a poussé :

Heikki Linnakangas a poussé :

  • Fix the same-rel optimization when creating WAL records. prev_regbuf was never set, and therefore the same-rel flag was never set on WAL records. Report and fix by Zhanq Zq http://git.postgresql.org/pg/commitdiff/ec3d976bce7e322c29f1007d19b63b7a3a1a6ee4
  • At promotion, archive last segment from old timeline with .partial suffix. Previously, we would archive the possible-incomplete WAL segment with its normal filename, but that causes trouble if the server owning that timeline is still running, and tries to archive the same segment later. It's not nice for the standby to trip up the master's archival like that. And it's pretty confusing, anyway, to have an incomplete segment in the archive that's indistinguishable from a normal, complete segment. To avoid such confusion, add a .partial suffix to the file. Or to be more precise, make a copy of the old segment under the .partial suffix, and archive that instead of the original file. pg_receivexlog also uses the .partial suffix for the same purpose, to tell apart incompletely streamed files from complete ones. There is no automatic mechanism to use the .partial files at recovery, so they will go unused, unless the administrator manually copies to them to the pg_xlog directory (and removes the .partial suffix). Recovery won't normally need the WAL - when recovering to the new timeline, it will find the same WAL on the first segment on the new timeline instead - but it nevertheless feels better to archive the file with the .partial suffix, for debugging purposes if nothing else. http://git.postgresql.org/pg/commitdiff/de7688442f5aaa03da60416a6aa3474738718803
  • Add macros to check if a filename is a WAL segment or other such file. We had many instances of the strlen + strspn combination to check for that. This makes the code a bit easier to read. http://git.postgresql.org/pg/commitdiff/179cdd098196338880bdbb39c39a788abdad4dd8

Robert Haas a poussé :

  • Use outerPlanState macro instead of referring to leffttree. This makes the executor code more consistent. It also removes an apparently superfluous NULL test in nodeGroup.c. Qingqing Zhou, reviewed by Tom Lane, and further revised by me. http://git.postgresql.org/pg/commitdiff/40f42d2a34329b0b71a1287d6fd2554298dbb713
  • Recursively fsync() the data directory after a crash. Otherwise, if there's another crash, some writes from after the first crash might make it to disk while writes from before the crash fail to make it to disk. This could lead to data corruption. Back-patch to all supported versions. Abhijit Menon-Sen, reviewed by Andres Freund and slightly revised by me. http://git.postgresql.org/pg/commitdiff/2ce439f3379aed857517c8ce207485655000fc8e
  • Fix some problems with patch to fsync the data directory. pg_win32_is_junction() was a typo for pgwin32_is_junction(). open() was used not only in a two-argument form, which breaks on Windows, but also where BasicOpenFile() should have been used. Per reports from Andrew Dunstan and David Rowley. http://git.postgresql.org/pg/commitdiff/456ff0863851d70dce679ca3f631392589e31a33
  • Avoid using a C++ keyword as a structure member name. Per request from Peter Eisentraut. http://git.postgresql.org/pg/commitdiff/1998261034ec7a948bb9b25b7cb88d014d371da1
  • Fix incorrect math in DetermineSafeOldestOffset. The old formula didn't have enough parentheses, so it would do the wrong thing, and it used / rather than % to find a remainder. The effect of these oversights is that the stop point chosen by the logic introduced in commit b69bf30b9bfacafc733a9ba77c9587cf54d06c0c might be rather meaningless. Thomas Munro, reviewed by Kevin Grittner, with a whitespace tweak by me. http://git.postgresql.org/pg/commitdiff/7be47c56af3d3013955c91c2877c08f2a0e3e6a2
  • Teach autovacuum about multixact member wraparound. The logic introduced in commit b69bf30b9bfacafc733a9ba77c9587cf54d06c0c and repaired in commits 669c7d20e6374850593cb430d332e11a3992bbcf and 7be47c56af3d3013955c91c2877c08f2a0e3e6a2 helps to ensure that we don't overwrite old multixact member information while it is still needed, but a user who creates many large multixacts can still exhaust the member space (and thus start getting errors) while autovacuum stands idly by. To fix this, progressively ramp down the effective value (but not the actual contents) of autovacuum_multixact_freeze_max_age as member space utilization increases. This makes autovacuum more aggressive and also reduces the threshold for a manual VACUUM to perform a full-table scan. This patch leaves unsolved the problem of ensuring that emergency autovacuums are triggered even when autovacuum=off. We'll need to fix that via a separate patch. Thomas Munro and Robert Haas http://git.postgresql.org/pg/commitdiff/53bb309d2d5a9432d2602c93ed18e58bd2924e15

Peter Eisentraut a poussé :

Álvaro Herrera a poussé :

  • Add geometry/range functions to support BRIN inclusion. This commit adds the following functions: box(point) -> box, bound_box(box, box) -> box, inet_same_family(inet, inet) -> bool, inet_merge(inet, inet) -> cidr, range_merge(anyrange, anyrange) -> anyrange. The first of these is also used to implement a new assignment cast from point to box. These functions are the first part of a base to implement an "inclusion" operator class for BRIN, for multidimensional data types. Author: Emre Hasegeli. Reviewed by: Andreas Karlsson http://git.postgresql.org/pg/commitdiff/3b6db1f445e14bd189ebc99ce1e5535a1c624613
  • Improve BRIN infra, minmax opclass and regression test. The minmax opclass was using the wrong support functions when cross-datatypes queries were run. Instead of trying to fix the pg_amproc definitions (which apparently is not possible), use the already correct pg_amop entries instead. This requires jumping through more hoops (read: extra syscache lookups) to obtain the underlying functions to execute, but it is necessary for correctness. Author: Emre Hasegeli, tweaked by Álvaro Review: Andreas Karlsson Also change BrinOpcInfo to record each stored type's typecache entry instead of just the OID. Turns out that the full type cache is necessary in brin_deform_tuple: the original code used the indexed type's byval and typlen properties to extract the stored tuple, which is correct in Minmax; but in other implementations that want to store something different, that's wrong. The realization that this is a bug comes from Emre also, but I did not use his patch. I also adopted Emre's regression test code (with smallish changes), which is more complete. http://git.postgresql.org/pg/commitdiff/db5f98ab4fa44bc563ec62d7b1aada4fc276d9b2

Magnus Hagander a poussé :

Bruce Momjian a poussé :

Andres Freund a poussé :

  • Represent columns requiring insert and update privileges indentently. Previously, relation range table entries used a single Bitmapset field representing which columns required either UPDATE or INSERT privileges, despite the fact that INSERT and UPDATE privileges are separately cataloged, and may be independently held. As statements so far required either insert or update privileges but never both, that was sufficient. The required permission could be inferred from the top level statement run. The upcoming INSERT ... ON CONFLICT UPDATE feature needs to independently check for both privileges in one statement though, so that is not sufficient anymore. Bumps catversion as stored rules change. Author: Peter Geoghegan. Reviewed-By: Andres Freund. http://git.postgresql.org/pg/commitdiff/2c8f4836db058d0715bc30a30655d646287ba509
  • Remove dependency on ordering in logical decoding upsert test. Buildfarm member magpie sorted the output differently than intended by Peter. "Resolve" the problem by simply not aggregating, it's not that many lines. http://git.postgresql.org/pg/commitdiff/581f4f9657fc3ab08199d02c0a4ea89c658882a6
  • Add support for INSERT ... ON CONFLICT DO NOTHING/UPDATE. The newly added ON CONFLICT clause allows to specify an alternative to raising a unique or exclusion constraint violation error when inserting. ON CONFLICT refers to constraints that can either be specified using a inference clause (by specifying the columns of a unique constraint) or by naming a unique or exclusion constraint. DO NOTHING avoids the constraint violation, without touching the pre-existing row. DO UPDATE SET ... [WHERE ...] updates the pre-existing tuple, and has access to both the tuple proposed for insertion and the existing tuple; the optional WHERE clause can be used to prevent an update from being executed. The UPDATE SET and WHERE clauses have access to the tuple proposed for insertion using the "magic" EXCLUDED alias, and to the pre-existing tuple using the table name or its alias. This feature is often referred to as upsert. This is implemented using a new infrastructure called "speculative insertion". It is an optimistic variant of regular insertion that first does a pre-check for existing tuples and then attempts an insert. If a violating tuple was inserted concurrently, the speculatively inserted tuple is deleted and a new attempt is made. If the pre-check finds a matching tuple the alternative DO NOTHING or DO UPDATE action is taken. If the insertion succeeds without detecting a conflict, the tuple is deemed inserted. To handle the possible ambiguity between the excluded alias and a table named excluded, and for convenience with long relation names, INSERT INTO now can alias its target table. Bumps catversion as stored rules change. Author: Peter Geoghegan, with significant contributions from Heikki Linnakangas and Andres Freund. Testing infrastructure by Jeff Janes. Reviewed-By: Heikki Linnakangas, Andres Freund, Robert Haas, Simon Riggs, Dean Rasheed, Stephen Frost and many others. http://git.postgresql.org/pg/commitdiff/168d5805e4c08bed7b95d351bf097cff7c07dd65
  • Minor ON CONFLICT related comments and doc fixes. Geoff Winkless, Stephen Frost, Peter Geoghegan and me. http://git.postgresql.org/pg/commitdiff/e8898e9169c851c2b8c98f981c1c4755a5758f8e
  • Fix two problems in infer_arbiter_indexes(). The first is a pretty simple bug where a relcache entry is used after the relation is closed. In this particular situation it does not appear to have bad consequences unless compiled with RELCACHE_FORCE_RELEASE. The second is that infer_arbiter_indexes() skipped indexes that aren't yet valid according to indcheckxmin. That's not required here, because uniqueness checks don't care about visibility according to an older snapshot. While thats not really a bug, it makes things undesirably non-deterministic. There is some hope that this explains a test failure on buildfarm member jaguarundi. Discussion: 9096.1431102730@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/bab64ef9e8bc56fa5db9bd41cefb54c3d8051dbe

Stephen Frost a poussé :

  • Remove reference to src/tools/backend/index.html. src/tools/backend was removed back in 63f1ccd, but backend/storage/lmgr/README didn't get the memo. Author: Amit Langote http://git.postgresql.org/pg/commitdiff/195fbd40123b85ba8a44ca273b17d699e30ec6a8
  • Add pg_file_settings view and function. The function and view added here provide a way to look at all settings in postgresql.conf, any #include'd files, and postgresql.auto.conf (which is what backs the ALTER SYSTEM command). The information returned includes the configuration file name, line number in that file, sequence number indicating when the parameter is loaded (useful to see if it is later masked by another definition of the same parameter), parameter name, and what it is set to at that point. This information is updated on reload of the server. This is unfiltered, privileged, information and therefore access is restricted to superusers through the GRANT system. Author: Sawada Masahiko, various improvements by me. Reviewers: David Steele http://git.postgresql.org/pg/commitdiff/a97e0c3354ace5d74c6873cd5e98444757590be8
  • Bump catversion for pg_file_settings. Pointed out by Andres (thanks!) Apologies for not including it in the initial patch. http://git.postgresql.org/pg/commitdiff/4b342fb591ebb556cab18fc999c8710e9733e5bb
  • Modify pg_stat_get_activity to build a tuplestore. This updates pg_stat_get_activity() to build a tuplestore for its results instead of using the old-style multiple-call method. This simplifies the function, though that wasn't the primary motivation for the change, which is that we may turn it into a helper function which can filter the results (or not) much more easily. http://git.postgresql.org/pg/commitdiff/f91feba8776eb66008cdb73b3a8c0c7c08cc54d9
  • Change default for include_realm to 1. The default behavior for GSS and SSPI authentication methods has long been to strip the realm off of the principal, however, this is not a secure approach in multi-realm environments and the use-case for the parameter at all has been superseded by the regex-based mapping support available in pg_ident.conf. Change the default for include_realm to be '1', meaning that we do NOT remove the realm from the principal by default. Any installations which depend on the existing behavior will need to update their configurations (ideally by leaving include_realm set to 1 and adding a mapping in pg_ident.conf, but alternatively by explicitly setting include_realm=0 prior to upgrading). Note that the mapping capability exists in all currently supported versions of PostgreSQL and so this change can be done today. Barring that, existing users can update their configurations today to explicitly set include_realm=0 to ensure that the prior behavior is maintained when they upgrade. This needs to be noted in the release notes. Per discussion with Magnus and Peter. http://git.postgresql.org/pg/commitdiff/9a0884176fdfa51551d6a3b26fa0e1b216c3e4c2
  • Improve ParseConfigFp comment wrt head/tail. The head_p and tail_p pointers passed to ParseConfigFp() are actually input/output parameters, not strictly output paramaters. This updates the function comment to reflect that. Per discussion with Tom. http://git.postgresql.org/pg/commitdiff/0cf56f14dd15532fec930b502cb6457023b01ef8
  • Correct reindexdb documentation. --schema takes a schema, not a table. Author: Sawada Masahiko http://git.postgresql.org/pg/commitdiff/f0a4b20bb9f91bdc0d60ff8732ee0195b0dfdd73

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in a patch to fix a potential pointer dereference in plperl.c caused by transforms patch.
  • Petr Korobeinikov sent in a patch to add \ev and \sv to edit and show the creation scripts for views, respectively, to psql.
  • Álvaro Herrera sent in two more revisions of a patch to add deparsing utilities.
  • Arjen Nienhuis sent in a patch to have GB18030 handle more than 2-byte Unicode code points.
  • Fabien COELHO sent in two more revisions of a patch to extend pgbench expressions with functions.
  • Emre Hasegeli and Álvaro Herrera traded patches to add a BRIN range operator class.
  • Tom Lane sent in another revision of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Tomas Vondra sent in another revision of a patch to add multivariate statistics.
  • Fabien COELHO sent in a patch to pgbench to allow '=' in \set.
  • SAWADA Masahiko and Fabrízio de Royes Mello traded patches to add REINDEX ... VERBOSE.
  • Stephen Frost sent in two more revisions of a patch to add default roles for sets of administrative functions.
  • Fabien COELHO sent in another revision of a patch to remove nclients/nthreads constraint from pgbench.
  • Stas Kelvich sent in another revision of a patch to add kNN support for the cube extension.
  • Michael Paquier sent in a patch to redefine subxcnt as uint32 for consistency with xcnt.
  • Michael Paquier sent in a patch to make more precise the rounding behavior of numeric and double precision in docs.
  • Kaigai Kouhei sent in three more revisions of a patch to add a custom/foreign join API.
  • Kaigai Kouhei sent in a patch to fix regtest policy for sepgsql.
  • Tom Lane sent in a patch to fix a bug in HOT.

par N Bougain le lundi 11 mai 2015 à 00h49

Nouvelles hebdomadaires de PostgreSQL - 3 mai 2015

La classe Consistent State's PG Admin aura lieu à Denver du 18 au 21 mai. Les inscriptions sont ouvertes jusqu'au 14 : http://www.consistentstate.com/store/c1/Featured_Products.html

PGConf Silicon Valley 2015 se tiendra au centre de convention sud de San Francisco les 17 & 18 novembre. L'appel à conférenciers porte jusqu'au 15 juin : http://www.pgconfsv.com

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mai

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Andrew Dunstan a poussé :

  • fix escaping of brackets for m4 broken in b6b2149e48aa61981ae0199c963d5145a37c258c http://git.postgresql.org/pg/commitdiff/f707b53449f3ab6998c615b746ad01f5775723a3
  • Enable transforms modules to build and run with Mingw builds. These modules were all missing essential Windows scaffolding, including resources files and descriptions, and links to the relevant library import files. This latter item means that the modules can't be built with pgxs on Windows, as we don't install the import files. If we ever decide to install them this restriction could probably be removed. Also, as with plperl we need to make sure that perl's CORE directory is last on the include list, as on Windows it appears to contain some headers with names that clash with names of some headers we include. http://git.postgresql.org/pg/commitdiff/f802c6ddba143bd88512b5fc34e84ae0b4883284
  • Fix python_includespec on Windows at configure time. By converting to using forward slashes at configure time we avoid having to repeat the logic anywhere that this is needed, such as in transforms modules for plpython. http://git.postgresql.org/pg/commitdiff/b6b2149e48aa61981ae0199c963d5145a37c258c
  • Make hstore_plperl's build more like plperl's. This involves moving perl's CORE library to the end of the include list, and adding other compilation settings that plperl uses. This won't completely fix the breakage currently being seen by gcc builds on Windows, but it will let the build get further, and should be wholly benign, if not beneficial, on *nix. http://git.postgresql.org/pg/commitdiff/77477e745be534c5925cf7cb8b9c6a7698c575a3
  • Enable transforms tests for python 2 on MSVC builds. Currently regression tests for python 3 are disabled on MSVC, and these tests fail with python 3, too, so we have some work to do to enable both. Meanwhile, all the buildfarm hosts seem to be building with python 2 anyway, so this at least gets us some coverage. Original patch from Michael Paquier, significantly modified by me. http://git.postgresql.org/pg/commitdiff/eb010637dd47be65e5d8b22d6965c2e96f6937b8
  • Fix MSVC builds for contrib transforms modules. With this patch the MSVC build and installation will work correctly with the transforms. However the python transform tests for hstore and ltree are still disabled pending some further adjustments. Michael Paquier with some tweaks from me. http://git.postgresql.org/pg/commitdiff/cbf9f0ec312e54481e777ffbe941b548d909081e
  • Fix vcbuild failures and chkpass dependency caused by 854adb8. Switching the Windows build scripts to use forward slashes instead of backslashes has caused a couple of issues in VC builds: - The file tree list was not correctly generated, build script generating vcproj file missing tree dependencies when listing items in Filter. - VC builds do not accept file paths with forward slashes, perhaps it could be possible to use a Condition but it seems safer to simply enforce the file paths to use backslashes in the vcproj files. - chkpass had an unneeded dependency with libpgport and libpgcommon to make build succeed but actually it is not necessary as crypt.c is already listed for this project and should be replaced with a fake name as it is a unique file. Michael Paquier http://git.postgresql.org/pg/commitdiff/06ca28d5ab2f810ef25e718e0d71f2233542c151

Stephen Frost a poussé :

  • Improve qual pushdown for RLS and SB views. The original security barrier view implementation, on which RLS is built, prevented all non-leakproof functions from being pushed down to below the view, even when the function was not receiving any data from the view. This optimization improves on that situation by, instead of checking strictly for non-leakproof functions, it checks for Vars being passed to non-leakproof functions and allows functions which do not accept arguments or whose arguments are not from the current query level (eg: constants can be particularly useful) to be pushed down. As discussed, this does mean that a function which is pushed down might gain some idea that there are rows meeting a certain criteria based on the number of times the function is called, but this isn't a particularly new issue and the documentation in rules.sgml already addressed similar covert-channel risks. That documentation is updated to reflect that non-leakproof functions may be pushed down now, if they meet the above-described criteria. Author: Dean Rasheed, with a bit of rework to make things clearer, along with comment and documentation updates from me. http://git.postgresql.org/pg/commitdiff/dcbf5948e12aa60b4d6ab65b6445897dfc971e01

Andres Freund a poussé :

  • Use a fd opened for read/write when syncing slots during startup. Some operating systems, including the reporter's windows, return EBADFD or similar when fsync() is invoked on a O_RDONLY file descriptor. Unfortunately RestoreSlotFromDisk() does exactly that; which causes failures after restarts in at least some scenarios. If you hit the bug the error message will be something like ERROR: could not fsync file "pg_replslot/$name/state": Bad file descriptor Simply use O_RDWR instead of O_RDONLY when opening the relevant file descriptor to fix the bug. Unfortunately I have no way of verifying the fix, but we've seen similar problems in the past. This bug goes back to 9.4 where slots were introduced. Backpatch accordingly. Reported-By: Patrice Drolet Bug: #13143: Discussion: 20150424101006.2556.60897@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/dfbaed459754e71e01bb0cc90a12802bba3f9786
  • Correct replication origin's use of UINT16_MAX to PG_UINT16_MAX. We can't rely on UINT16_MAX being present, which is why we introduced PG_UINT16_MAX... Buildfarm animal bowerbird via Andrew Gierth. http://git.postgresql.org/pg/commitdiff/e0f26fc76532defd06caf79d711fa99cea83c532
  • Fix unaligned memory access in xlog parsing due to replication origin patch. ParseCommitRecord() accessed xl_xact_origin directly. But the chunks in the commit record's data only have 4 byte alignment, whereas xl_xact_origin's members require 8 byte alignment on some platforms. Update comments to make not of that and copy the record to stack local storage before reading. With help from Stefan Kaltenbrunner in pinning down the buildfarm and verifying the fix. http://git.postgresql.org/pg/commitdiff/1db12da85bee7a01abfbbf2798904347e4d9515a
  • Introduce replication progress tracking infrastructure. When implementing a replication solution ontop of logical decoding, two related problems exist: * How to safely keep track of replication progress * How to change replication behavior, based on the origin of a row; e.g. to avoid loops in bi-directional replication setups The solution to these problems, as implemented here, consist out of three parts: 1) 'replication origins', which identify nodes in a replication setup. 2) 'replication progress tracking', which remembers, for each replication origin, how far replay has progressed in a efficient and crash safe manner. 3) The ability to filter out changes performed on the behest of a replication origin during logical decoding; this allows complex replication topologies. E.g. by filtering all replayed changes out. Most of this could also be implemented in "userspace", e.g. by inserting additional rows contain origin information, but that ends up being much less efficient and more complicated. We don't want to require various replication solutions to reimplement logic for this independently. The infrastructure is intended to be generic enough to be reusable. This infrastructure also replaces the 'nodeid' infrastructure of commit timestamps. It is intended to provide all the former capabilities, except that there's only 2^16 different origins; but now they integrate with logical decoding. Additionally more functionality is accessible via SQL. Since the commit timestamp infrastructure has also been introduced in 9.5 (commit 73c986add) changing the API is not a problem. For now the number of origins for which the replication progress can be tracked simultaneously is determined by the max_replication_slots GUC. That GUC is not a perfect match to configure this, but there doesn't seem to be sufficient reason to introduce a separate new one. Bumps both catversion and wal page magic. Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer Discussion: 20150216002155.GI15326@awork2.anarazel.de, 20140923182422.GA15776@alap3.anarazel.de, 20131114172632.GE7522@alap2.anarazel.de http://git.postgresql.org/pg/commitdiff/5aa2350426c4fdb3d04568b65aadac397012bbcb
  • Copy editing of the replication origins patch. Michael Paquier and myself. http://git.postgresql.org/pg/commitdiff/2b22795b32576fa7173b501b646581a17de35902

Álvaro Herrera a poussé :

  • Protect against multixact members wraparound. Multixact member files are subject to early wraparound overflow and removal: if the average multixact size is above a certain threshold (see note below) the protections against offset overflow are not enough: during multixact truncation at checkpoint time, some pg_multixact/members files would be removed because the server considers them to be old and not needed anymore. This leads to loss of files that are critical to interpret existing tuples's Xmax values. To protect against this, since we don't have enough info in pg_control and we can't modify it in old branches, we maintain shared memory state about the oldest value that we need to keep; we use this during new multixact creation to abort if an old still-needed file would get overwritten. This value is kept up to date by checkpoints, which makes it not completely accurate but should be good enough. We start emitting warnings sometime earlier, so that the eventual multixact-shutdown doesn't take DBAs completely by surprise (more precisely: once 20 members SLRU segments are remaining before shutdown.) On troublesome average multixact size: The threshold size depends on the multixact freeze parameters. The oldest age is related to the greater of multixact_freeze_table_age and multixact_freeze_min_age: anything older than that should be removed promptly by autovacuum. If autovacuum is keeping up with multixact freezing, the troublesome multixact average size is (2^32-1) / Max(freeze table age, freeze min age) or around 28 members per multixact. Having an average multixact size larger than that will eventually cause new multixact data to overwrite the data area for older multixacts. (If autovacuum is not able to keep up, or there are errors in vacuuming, the actual maximum is multixact_freeeze_max_age instead, at which point multixact generation is stopped completely. The default value for this limit is 400 million, which means that the multixact size that would cause trouble is about 10 members). Initial bug report by Timothy Garnett, bug #12990 Backpatch to 9.3, where the problem was introduced. Authors: Álvaro Herrera, Thomas Munro Reviews: Thomas Munro, Amit Kapila, Robert Haas, Kevin Grittner http://git.postgresql.org/pg/commitdiff/b69bf30b9bfacafc733a9ba77c9587cf54d06c0c
  • Code review for multixact bugfix. Reword messages, rename a confusingly named function. Per Robert Haas. http://git.postgresql.org/pg/commitdiff/d3821e70c9b6d76083f4eb0f4cc25716e961c89d
  • Fix pg_upgrade's multixact handling (again) We need to create the pg_multixact/offsets file deleted by pg_upgrade much earlier than we originally were: it was in TrimMultiXact(), which runs after we exit recovery, but it actually needs to run earlier than the first call to SetMultiXactIdLimit (before recovery), because that routine already wants to read the first offset segment. Per pg_upgrade trouble report from Jeff Janes. While at it, silence a compiler warning about a pointless assert that an unsigned variable was being tested non-negative. This was a signed constant in Thomas Munro's patch which I changed to unsigned before commit. Pointed out by Andres Freund. http://git.postgresql.org/pg/commitdiff/669c7d20e6374850593cb430d332e11a3992bbcf
  • Fix up some loose ends for CURRENT_USER as RoleSpec. In commit 31eae6028eca4, some documents were not updated to show the new capability; fix that. Also, the error message you get when CURRENT_USER and SESSION_USER are used in a context that doesn't accept them could be clearer about it being a problem only in those contexts; so add the word "here". Author: Kyotaro HORIGUCHI His patch submission also included changes to GRANT/REVOKE, but those seemed more controversial, so I left them out. We can reconsider these changes later. http://git.postgresql.org/pg/commitdiff/9d396af46357df1243aff4a9ca4f4987e4fe6024

Tom Lane a poussé :

  • Fix ATSimpleRecursion() to allow recursion from a foreign table. This is necessary in view of the changes to allow foreign tables to be full members of inheritance hierarchies, but I (tgl) unaccountably missed it in commit cb1ca4d800621dcae67ca6c799006de99fa4f0a5. Noted by Amit Langote, patch by Etsuro Fujita http://git.postgresql.org/pg/commitdiff/ad9f08f70636051b5d5fe8d57062994b7335a960
  • Fix another test for RELKIND_RELATION that should allow foreign tables now. I thought I'd gone through all of these before, but a fresh review found this one too. (Perhaps it would be better to just delete this test and let the failure occur later, but for the moment I'll preserve the logic.) The case that this was rejecting is like CREATE FOREIGN TABLE ft (f1 int ...) ...; CREATE TABLE c1 (UNIQUE(f1)) INHERITS(ft); http://git.postgresql.org/pg/commitdiff/290713e31a1ee04eed7877985a4c28a30fd0d1db
  • Fix overlooked relcache invalidation in ALTER TABLE ... ALTER CONSTRAINT. When altering the deferredness state of a foreign key constraint, we correctly updated the catalogs and then invalidated the relcache state for the target relation ... but that's not the only relation with relevant triggers. Must invalidate the other table as well, or the state change fails to take effect promptly for operations triggered on the other table. Per bug #13224 from Christian Ullrich. In passing, reorganize regression test case for this feature so that it isn't randomly injected into the middle of an unrelated test sequence. Oversight in commit f177cbfe676dc2c7ca2b206c54d6bf819feeea8b. Back-patch to 9.4 where the faulty code was added. http://git.postgresql.org/pg/commitdiff/a4820434c1a62e0c5f4051a31ad8b4a11f0a6ad7

Bruce Momjian a poussé :

Robert Haas a poussé :

  • psql: Improve tab completion for ALTER FOREIGN TABLE. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/c6e96a2f986e4dad72c14b14d4cc17d02b2a6aad
  • Attempt to fix some compiler warnings. http://git.postgresql.org/pg/commitdiff/ef3f9e642d2b2bba8933b4cff4039ce0d354ce08
  • Remove enum-related special cases for catalog scans. When this code was written, catalog scans were normally performed using SnapshotNow, making special handling necessary here. Now, however, all catalog scans use MVCC snapshots, so we can change these cases to look more like what we do for catalog scans elsewhere in the code. Per discussion with Tom Lane and a reminder from Bruce Momjian. http://git.postgresql.org/pg/commitdiff/9b6a0ce5f060315f900de7b398d5197a2e42f2f2
  • Update .gitignore for new rmgr, changed paths. http://git.postgresql.org/pg/commitdiff/fe72c4c55bd65772aa5a4130309150ffca1749ab
  • Add <literal> markup, for consistency. This file isn't entirely consistent about whether "on" and "off" should be marked up with <literal>, but it doesn't make much sense to be inconsistent within a single sentence. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/49601ab1a6df9059f7000f78af43f2e74afefa59
  • Create an infrastructure for parallel computation in PostgreSQL. This does four basic things. First, it provides convenience routines to coordinate the startup and shutdown of parallel workers. Second, it synchronizes various pieces of state (e.g. GUCs, combo CID mappings, transaction snapshot) from the parallel group leader to the worker processes. Third, it prohibits various operations that would result in unsafe changes to that state while parallelism is active. Finally, it propagates events that would result in an ErrorResponse, NoticeResponse, or NotifyResponse message being sent to the client from the parallel workers back to the master, from which they can then be sent on to the client. Robert Haas, Amit Kapila, Noah Misch, Rushabh Lathia, Jeevan Chalke. Suggestions and review from Andres Freund, Heikki Linnakangas, Noah Misch, Simon Riggs, Euler Taveira, and Jim Nasby. http://git.postgresql.org/pg/commitdiff/924bcf4f16d54c55310b28f77686608684734f42
  • Allow FDWs and custom scan providers to replace joins with scans. Foreign data wrappers can use this capability for so-called "join pushdown"; that is, instead of executing two separate foreign scans and then joining the results locally, they can generate a path which performs the join on the remote server and then is scanned locally. This commit does not extend postgres_fdw to take advantage of this capability; it just provides the infrastructure. Custom scan providers can use this in a similar way. Previously, it was only possible for a custom scan provider to scan a single relation. Now, it can scan an entire join tree, provided of course that it knows how to produce the same results that the join would have produced if executed normally. KaiGai Kohei, reviewed by Shigeru Hanada, Ashutosh Bapat, and me. http://git.postgresql.org/pg/commitdiff/e7cb7ee14555cc9c5773e2c102efd6371f6f2005
  • Deparse named arguments to use the new => operator instead of :=. Tom Lane pointed out that this wasn't done, and asked whether that was intentional. Subsequent discussion was in favor of making the change, so here we go. http://git.postgresql.org/pg/commitdiff/e044a44949c5b9c9f548e8a1cd4bf0b50fb2a1cc

Peter Eisentraut a poussé :

  • Fix parallel make risk with new check temp-install setup. The "check" target no longer needs to depend on "all", because it now runs "install" directly, which in turn depends on "all". Doing both will cause problems with parallel make, because two builds will run next to each other. Also remove the redirection of the temp-install output into a log file. This was appropriate when this was done from within pg_regress, but now it's just a regular make run, and especially with the above changes this will now take the place of running the "all" target before the test suites. problem report by Jeff Janes, patch in part by Michael Paquier http://git.postgresql.org/pg/commitdiff/dbf2ec1a1c053379e2f9a5913979a1ca4dccbd43
  • Fix shared libpython detection on OS X. Apparently, looking for an appropriately named file doesn't work on some older versions, so put the back the explicit platform detection. http://git.postgresql.org/pg/commitdiff/010aa420b9dd5393490c8793f43949dca269a753
  • hstore_plperl: Move port-specific parts to later in the makefile. PORTNAME isn't set until the global makefiles have been included. http://git.postgresql.org/pg/commitdiff/e30a864963c5f98bafeaf3005f258ff90916da7c
  • Windows also needs an override of the shared libpython detection http://git.postgresql.org/pg/commitdiff/67df9782e94765a281a82a7672d39ebb4bf58828
  • Move interpreter shared library detection to configure. For building PL/Perl, PL/Python, and PL/Tcl, we need a shared library of libperl, libpython, and libtcl, respectively. Previously, this was checked in the makefiles, skipping the PL build with a warning if no shared library was available. Now this is checked in configure, with an error if no shared library is available. The previous situation arose because in the olden days, the configure options --with-perl, --with-python, and --with-tcl controlled whether frontend interfaces for those languages would be built. The procedural languages were added later, and shared libraries were often not available in the beginning. So it was decided skip the builds of the procedural languages in those cases. The frontend interfaces have since been removed from the tree, and shared libraries are now available most of the time, so that setup makes much less sense now. Also, the new setup allows contrib modules and pgxs users to rely on the respective PLs being available based on configure flags. http://git.postgresql.org/pg/commitdiff/d664a10f9623fd2198b257e513bce849d439a773
  • Make hstore_plperl's build even more like plperl's. Combine the two places that set CPPFLAGS into one. Also, some settings should be restricted to Windows only. More precisely, -Wno-comment is a GCC-only option, but Windows in a makefile implies GCC at the moment. Also, since -Wno-comment is more properly a preprocessor option, move it to CPPFLAGS to simplify things a bit. http://git.postgresql.org/pg/commitdiff/0fd764647a9910a340359bb319929b70317b2ae4

Magnus Hagander a poussé :

Heikki Linnakangas a poussé :

  • Fix pg_rewind regression failure after "fast promotion". pg_rewind looks at the control file to determine the server's timeline. If the standby performs a "fast promotion", the timeline ID in the control file is not updated until the next checkpoint. The startup process requests a checkpoint immediately after promotion, so this is unlikely to be an issue in the real world, but the regression suite ran pg_rewind so quickly after promotion that the checkpoint had not yet completed. Reported by Stephen Frost http://git.postgresql.org/pg/commitdiff/484a848a73fc5a76c16bc73626b290154b6a57b4

Noah Misch a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Etsuro Fujita sent in two revisions of a patch to allow postgres_fdw option to control whether check constraints on remote tables are included in the definitions of foreign tables imported from a remote PG server when performing IMPORT FOREIGN SCHEMA.
  • Fabien COELHO sent in a patch to remove nclients/nthreads constraint from pgbench.
  • Petr Jelinek sent in another revision of a patch to implement a sequence access method.
  • Stephen Frost sent in another revision of a patch to change default for include_realm to 0.
  • Fabien COELHO sent in a patch to remove the pgbench thread fork-emulation code and simplify things where possible, especially around pthread_create and pthread_join.
  • Jim Nasby sent in a patch to allow SQL/plpgsql functions to accept record.
  • Bruce Momjian sent in two more revisions of a patch to make tables created by CREATE TABLE (...LIKE...) have OIDs if any of its parents did.
  • SAWADA Masahiko sent in six more revisions of a patch to create a pg_file_settings view.
  • Pavel Stehule sent in two more revisions of a patch to control context of RAISE in PL/pgsql.
  • Shigeru HANADA sent in another revision of a patch to add foreign joins.
  • Jeff Janes sent in a patch to support contrib extensions in the upcoming multivariate statistics patch.
  • Tomas Vondra sent in a patch to teach the expression walker about RestrictInfo.
  • Fabrízio de Royes Mello sent in a patch to implement COMMENT ON CURRENT DATABASE IS...
  • Jeff Janes sent in a patch to fix a compiler warning in the recent transforms feature.
  • Alexander Korotkov sent in a patch to add selectivity estimation functions to &&, @>, <@ operators in the intarray contrib extension.
  • David Rowley sent in a patch to fix some issues that came up with the new replication identifiers.
  • Petr Jelinek sent in another revision of a patch to add mogrify and indent features for the jsonb data type.
  • John Gorman sent in a patch to rectify some incompatible trig error handling.
  • Denis Kirjanov sent in a patch to enforce access control on security labels defined by admin and prohibit users from relabeling the objects.
  • Bruce Momjian sent in a patch to pg_upgrade to quote directory names in delete_old_cluster script.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add regrole.
  • Pavel Stehule sent in a patch to fix a bug in the implementation of errhidecontext().
  • SAWADA Masahiko sent in a patch to add a bit indicating frozen-ness to the visibility map.
  • Robert Haas sent in another revision of a patch to allow making vacuum failures not fatal in pgbench.
  • SAWADA Masahiko sent in two more revisions of a patch to implement REINDEX ... VERBOSE.
  • Qingqing Zhou sent in two more revisions of a patch to use outerPlanState() consistently in the executor code.
  • David Steele sent in another revision of a patch to implement pg_audit.
  • Abhijit Menon-Sen and Robert Haas traded patches to recursively fsync PGDATA at startup after a crash.
  • Bruce Momjian and Tom Lane traded patches to add a procost for to_tsvector.
  • Emre Hasegeli sent in another revision of a patch to add a BRIN range operator class.

par N Bougain le lundi 11 mai 2015 à 00h44

vendredi 1 mai 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 26 avril 2015

Le PGDay Campinas 2015 aura lieu à Campinas (Brésil) le 7 août. L'appel à conférenciers expire le 31 mai : http://pgdaycampinas.com.br/english/

L'appel à conférenciers pour la session PostgreSQL n°7 (24 septembre 2015 à Paris) est lancé jusqu'au 15 juin 2015 : call-for-paper <AT> postgresql-sessions <DOT> org. http://www.postgresql-sessions.org/7/about

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Peter Eisentraut a poussé :

Bruce Momjian a poussé :

Andres Freund a poussé :

Robert Haas a poussé :

Heikki Linnakangas a poussé :

Stephen Frost a poussé :

  • Pull in tableoid for inheiritance with rowMarks. As noted by Etsuro Fujita [1] and Dean Rasheed[2], cb1ca4d800621dcae67ca6c799006de99fa4f0a5 changed ExecBuildAuxRowMark() to always look for the tableoid in the target list, but didn't also change preprocess_targetlist() to always include the tableoid. This resulted in errors with soon-to-be-added RLS with inheritance tests, and errors when using inheritance with foreign tables. Authors: Etsuro Fujita and Dean Rasheed (independently) Minor word-smithing on the comments by me. [1] 552CF0B6.8010006@lab.ntt.co.jp [2] CAEZATCVmFUfUOwwhnBTcgi6AquyjQ0-1fyKd0T3xBWJvn+xsFA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/4ccc5bd28e7f0c0d1b221683398ae178515b9f76
  • RLS fixes, new hooks, and new test module. In prepend_row_security_policies(), defaultDeny was always true, so if there were any hook policies, the RLS policies on the table would just get discarded. Fixed to start off with defaultDeny as false and then properly set later if we detect that only the default deny policy exists for the internal policies. The infinite recursion detection in fireRIRrules() didn't properly manage the activeRIRs list in the case of WCOs, so it would incorrectly report infinite recusion if the same relation with RLS appeared more than once in the rtable, for example "UPDATE t ... FROM t ...". Further, the RLS expansion code in fireRIRrules() was handling RLS in the main loop through the rtable, which lead to RTEs being visited twice if they contained sublink subqueries, which prepend_row_security_policies() attempted to handle by exiting early if the RTE already had securityQuals. That doesn't work, however, since if the query involved a security barrier view on top of a table with RLS, the RTE would already have securityQuals (from the view) by the time fireRIRrules() was invoked, and so the table's RLS policies would be ignored. This is fixed in fireRIRrules() by handling RLS in a separate loop at the end, after dealing with any other sublink subqueries, thus ensuring that each RTE is only visited once for RLS expansion. The inheritance planner code didn't correctly handle non-target relations with RLS, which would get turned into subqueries during planning. Thus an update of the form "UPDATE t1 ... FROM t2 ..." where t1 has inheritance and t2 has RLS quals would fail. Fix by making sure to copy in and update the securityQuals when they exist for non-target relations. process_policies() was adding WCOs to non-target relations, which is unnecessary, and could lead to a lot of wasted time in the rewriter and the planner. Fix by only adding WCO policies when working on the result relation. Also in process_policies, we should be copying the USING policies to the WITH CHECK policies on a per-policy basis, fix by moving the copying up into the per-policy loop. Lastly, as noted by Dean, we were simply adding policies returned by the hook provided to the list of quals being AND'd, meaning that they would actually restrict records returned and there was no option to have internal policies and hook-based policies work together permissively (as all internal policies currently work). Instead, explicitly add support for both permissive and restrictive policies by having a hook for each and combining the results appropriately. To ensure this is all done correctly, add a new test module (test_rls_hooks) to test the various combinations of internal, permissive, and restrictive hook policies. Largely from Dean Rasheed (thanks!): CAEZATCVmFUfUOwwhnBTcgi6AquyjQ0-1fyKd0T3xBWJvn+xsFA@mail.gmail.com Author: Dean Rasheed, though I added the new hooks and test module. http://git.postgresql.org/pg/commitdiff/0bf22e0c8b1114ae37939c500535307abefd38e1
  • Fix installcheck for test_rls_hooks. As pointed out by the buildfarm, test_rls_hooks wasn't functioning properly with a clean installcheck. test_rls_hooks needs to explicitly load the library with the hooks in it, to allow installcheck to work; using the --temp-config doesn't help since that isn't used when running installcheck and it isn't exactly fair to the buildfarm to modify the installed config prior to calling installcheck. Also, have test_rls_hooks clean up after itself. http://git.postgresql.org/pg/commitdiff/450fa1b5ba0e986a20d8c017500c0c0bbf1e0b4b
  • Copy the relation name for error reporting in WCOs. In get_row_security_policies(), we need to make a copy of the relation name when building the WithCheckOptions structure, since RelationGetRelationName just returns a pointer into the local Relation structure. The relation name in the WCO structure is only used for error reporting. Pointed out by Robert and Christian Ullrich, who noted that the buildfarm members with -DCLOBBER_CACHE_ALWAYS were failing. http://git.postgresql.org/pg/commitdiff/cb087ec03bbb1d52845a4de83a6bf634dac2639f
  • Perform RLS WITH CHECK before constraints, etc. The RLS capability is built on top of the WITH CHECK OPTION system which was added for auto-updatable views, however, unlike WCOs on views (which are mandated by the SQL spec to not fire until after all other constraints and checks are done), it makes much more sense for RLS checks to happen earlier than constraint and uniqueness checks. This patch reworks the structure which holds the WCOs a bit to be explicitly either VIEW or RLS checks and the RLS-related checks are done prior to the constraint and uniqueness checks. This also allows better error reporting as we are now reporting when a violation is due to a WITH CHECK OPTION and when it's due to an RLS policy violation, which was independently noted by Craig Ringer as being confusing. The documentation is also updated to include a paragraph about when RLS WITH CHECK handling is performed, as there have been a number of questions regarding that and the documentation was previously silent on the matter. Author: Dean Rasheed, with some kabitzing and comment changes by me. http://git.postgresql.org/pg/commitdiff/e89bd02f58ac07e44e0388a32b7ee1b42f1fd7c6
  • Fix file comment for test_rls_hooks.c. The file-level comment wasn't updated when it was copied from the shared memory queue test module. Fixed. Noted by Dean Rasheed. http://git.postgresql.org/pg/commitdiff/410cbfd6dd778e8f388fd0d7ee9d84f833700da5

Álvaro Herrera a poussé :

  • Use the right type OID after creating a shell type. Commit a2e35b53c39b2a neglected to update the type OID to use further down in DefineType when TypeShellMake was changed to return ObjectAddress instead of OID (it got it right in DefineRange, however.) This resulted in an internal error message being issued when looking up I/O functions. Author: Michael Paquier. Also add Asserts() to a couple of other places to ensure that the type OID being used is as expected. http://git.postgresql.org/pg/commitdiff/50a16e30ebd76e70fc76abb2c8f0cd1e71deac41

Tom Lane a poussé :

  • Fix obsolete comment in set_rel_size(). The cross-reference to set_append_rel_pathlist() was obsoleted by commit e2fa76d80ba571d4de8992de6386536867250474, which split what had been set_rel_pathlist() and child routines into two sets of functions. But I (tgl) evidently missed updating this comment. Back-patch to 9.2 to avoid unnecessary divergence among branches. Amit Langote http://git.postgresql.org/pg/commitdiff/70d44dd9de2b781436ef1d55906614d241e02249
  • Fix up .gitignore and cleanup actions in some src/test/ subdirectories. examples/, locale/, and thread/ lacked .gitignore files and were also not connected up to top-level "make clean" etc. This had escaped notice because none of those directories are built in normal scenarios. Still, they have working Makefiles, so if someone does a "make" in one of these directories it would be good if (a) git doesn't bleat about the product files and (b) cleaning up removes them. This is a longstanding oversight, but since this behavior is probably only of interest to developers, there seems no need for back-patching. Michael Paquier and Tom Lane http://git.postgresql.org/pg/commitdiff/732b33f8ae4ecc9d7a9f07fd4cb74a60a2a5e2c2
  • Prevent improper reordering of antijoins vs. outer joins. An outer join appearing within the RHS of an antijoin can't commute with the antijoin, but somehow I missed teaching make_outerjoininfo() about that. In Teodor Sigaev's recent trouble report, this manifests as a "could not find RelOptInfo for given relids" error within eqjoinsel(); but I think silently wrong query results are possible too, if the planner misorders the joins and doesn't happen to trigger any internal consistency checks. It's broken as far back as we had antijoins, so back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/3cf8686014f91174018f20e01dbb0dafdcad0473
  • Add comments warning against generalizing default_with_oids. pg_dump has historically assumed that default_with_oids affects only plain tables and not other relkinds. Conceivably we could make it apply to some newly invented relkind if we did so from the get-go, but changing the behavior for existing object types will break existing dump scripts. Add code comments warning about this interaction. Also, make sure that default_with_oids doesn't cause parse_utilcmd.c to think that CREATE FOREIGN TABLE will create an OID column. I think this is only a latent bug right now, since we don't allow UNIQUE/PKEY constraints in CREATE FOREIGN TABLE, but it's better to be consistent and future-proof. http://git.postgresql.org/pg/commitdiff/0bd11d9711b88e72d2022e25b9227c480aca4978
  • Fix typo in linux startup script. Missed a "$" in what was meant to be a variable substitution. Careless mistake in commit f23425fa950fec3aff458de117037c9caadbc35c. http://git.postgresql.org/pg/commitdiff/f320cbb615e0374b18836337713239da58705cf3

Noah Misch a poussé :

Andrew Dunstan a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Jeff Janes sent in a patch to help make vacuum truncation scans faster.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT IGNORE (and UPDATE).
  • Michael Paquier sent in another revision of a patch to ensure that logical decoding follows timelines.
  • Heikki Linnakangas sent in another revision of a patch to implement access menthods for sequences.
  • Michael Paquier sent in two more revisions of a patch to support TAP tests with MSVC and Windows.
  • SAWADA Masahiko sent in another revision of a patch to help avoid freezing very large tables.
  • Etsuro Fujita sent in another revision of a patch to allow foreign tables to be created with OIDs.
  • Michael Paquier sent in another revision of a patch to ensure that Install.bat works when the target directory contains a space.
  • Michael Paquier sent in another revision of a patch to add pg_settings.pending_restart.
  • Kaigai Kouhei sent in another revision of a patch to add a custom join API.
  • Jan de Visser sent in another revision of a patch to let pg_ctl check the result of a postmaster config reload.
  • Petr Jelinek sent in a patch to add catalog regression tests for the TABLESAMPLE feature.
  • Michael Paquier sent in a patch to fix a comment in src/backend/access/transam/xlog.c.
  • Michael Paquier sent in a patch to allow releasing LWLocks on failure in cases where that makes sense.
  • Amit Kapila sent in another revision of a patch to implement parallel sequential scan.
  • Abhijit Menon-Sen sent in two more revisions of a patch to create a fast bloat measurement tool.
  • Fabrízio de Royes Mello sent in another revision of a patch to add CINE for ALTER TABLE ... ADD COLUMN.
  • David Steele sent in another revision of a patch to add pg_audit.
  • Kyotaro HORIGUCHI sent in two revisions of a patch to fix an issue where coerce_type() fails for Var of type UNKNOWNOID.
  • Andres Freund sent in another revision of a patch to implement replication identifiers.
  • Michael Paquier sent in a patch to fix broken collate.linux.utf8 test coverage.
  • Pavel Stehule sent in two more revisions of a patch to tune the displayability of context messages in PL/pgSQL errors.
  • Fabien COELHO sent in another revision of a patch to extend pgbench expressions with functions.
  • Fabien COELHO sent in another revision of a patch to fix pgbench --progress report under (very) low rate.
  • Euler Taveira de Oliveira sent in a patch to fix some issues in the parallel mode and parallel contexts.

par N Bougain le vendredi 1 mai 2015 à 14h17

samedi 25 avril 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 19 avril 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Peter Eisentraut a poussé :

Fujii Masao a poussé :

  • Silence gettext warning about '\r' escape sequence in translatable string. gettext was unhappy about the commit b216ad7 because it revealed the problem that internationalized messages may contain '\r' escape sequence in pg_rewind. This commit moves '\r' to a separate printf() call. Michael Paquier, bug reported by Peter Eisentraut http://git.postgresql.org/pg/commitdiff/1f94bec7a9e3e6b4fa5468236cf531fec16d1093

Heikki Linnakangas a poussé :

  • Refactor and fix TAP tests of pg_rewind * Don't pass arguments to prove, since that's not supported on perl 5.8 which is the minimum version supported by the TAP tests. Refactor the test files themselves to run the tests twice, in both local and remote mode. * Use eq rather than == for string comparison. This thinko caused the remote versions of the tests to never run. * Add "use strict" and "use warnings", and fix warnings that that produced. * Increase the delay after standby promotion, to make the tests more robust. * In remote mode, the connection string to the promoted standby was incorrect, leading to connection errors. Patch by Michael Paquier, to address Peter Eisentraut's report. http://git.postgresql.org/pg/commitdiff/53ba10770a315361770efdc17d2c01f6a30e3e3d
  • Fix pg_rewind regression tests in VPATH builds. Should call just "pg_rewind", instead of "./pg_rewind". The tests are called so that PATH contains the temporariy installation bin dir. Per report from Alvaro Herrera http://git.postgresql.org/pg/commitdiff/b22a36a62ce312c1df9477382d1da602b0c24f6f
  • Reorganize our CRC source files again. Now that we use CRC-32C in WAL and the control file, the "traditional" and "legacy" CRC-32 variants are not used in any frontend programs anymore. Move the code for those back from src/common to src/backend/utils/hash. Also move the slicing-by-8 implementation (back) to src/port. This is in preparation for next patch that will add another implementation that uses Intel SSE 4.2 instructions to calculate CRC-32C, where available. http://git.postgresql.org/pg/commitdiff/4f700bcd20c087f60346cb8aefd0e269be8e2157
  • Don't archive bogus recycled or preallocated files after timeline switch. After a timeline switch, we would leave behind recycled WAL segments that are in the future, but on the old timeline. After promotion, and after they become old enough to be recycled again, we would notice that they don't have a .ready or .done file, create a .ready file for them, and archive them. That's bogus, because the files contain garbage, recycled from an older timeline (or prealloced as zeros). We shouldn't archive such files. This could happen when we're following a timeline switch during replay, or when we switch to new timeline at end-of-recovery. To fix, whenever we switch to a new timeline, scan the data directory for WAL segments on the old timeline, but with a higher segment number, and remove them. Those don't belong to our timeline history, and are most likely bogus recycled or preallocated files. They could also be valid files that we streamed from the primary ahead of time, but in any case, they're not needed to recover to the new timeline. http://git.postgresql.org/pg/commitdiff/b2a5545bd63fc94a71b1e97ecdd03c605d97a438
  • Use Intel SSE 4.2 CRC instructions where available. Modern x86 and x86-64 processors with SSE 4.2 support have special instructions, crc32b and crc32q, for calculating CRC-32C. They greatly speed up CRC calculation. Whether the instructions can be used or not depends on the compiler and the target architecture. If generation of SSE 4.2 instructions is allowed for the target (-msse4.2 flag on gcc and clang), use them. If they are not allowed by default, but the compiler supports the -msse4.2 flag to enable them, compile just the CRC-32C function with -msse4.2 flag, and check at runtime whether the processor we're running on supports it. If it doesn't, fall back to the slicing-by-8 algorithm. (With the common defaults on current operating systems, the runtime-check variant is what you get in practice.) Abhijit Menon-Sen, heavily modified by me, reviewed by Andres Freund. http://git.postgresql.org/pg/commitdiff/3dc2d62d0486325bf263655c2d9a96aee0b02abe
  • Try to fix the CRC-32C autoconf magic for icc compiler. On gcc and clang, the _mm_crc32_u8 and _mm_crc32_u64 intrinsics are not defined at all, when not building with -msse4.2. But on icc, they are. So we cannot assume that if those intrinsics are defined, we can always use them safely, we might still need the runtime check. To fix, check if the __SSE_4_2__ preprocessor symbol is defined. That's supposed to be defined only when the compiler is targeting a processor that has SSE 4.2 support. Per buildfarm members fulmar and okapi. http://git.postgresql.org/pg/commitdiff/b4eb2d168d2c426978a02de8b9b6ccdb85e1b442
  • Oops, fix misspelled #endif. I hope this fixes the Windows builfarm failures. http://git.postgresql.org/pg/commitdiff/b73e7a0716264e5159947b1a755b9ab864142489
  • Optimize pg_comp_crc32c_sse42 routine slightly, and also use it on x86. Eliminate the separate 'len' variable from the loops, and also use the 4 byte instruction. This shaves off a few more cycles. Even though this routine that uses the special SSE 4.2 instructions is much faster than a generic routine, it's still a hot spot, so let's make it as fast as possible. Change the configure test to not test _mm_crc32_u64. That variant is only available in the 64-bit x86-64 architecture, not in 32-bit x86. Modify pg_comp_crc32c_sse42 so that it only uses _mm_crc32_u64 on x86-64. With these changes, the SSE accelerated CRC-32C implementation can also be used on 32-bit x86 systems. This also fixes the 32-bit MSVC build. http://git.postgresql.org/pg/commitdiff/936546dcbc24ad1f2b3d33e73aa5c5fde4d2be84
  • Fix logic to skip checkpoint if no records have been inserted. After the WAL format changes, the calculation of the size of a checkpoint record became incorrect. Instead of trying to fix the math, check that the previous record, i.e. the xl_prev value that we'd write for the next record, matches the last checkpoint's redo pointer. That way it's not dependent on the size of the checkpoint record at all. The old logic was actually slightly wrong all along: if the previous checkpoint record crossed a page boundary, the page headers threw off the record size calculation, and the checkpoint was not skipped. The new checkpoint would not cross a page boundary, so this only resulted in at most one extra checkpoint after the system became idle. The new logic fixes that. (It's not worth fixing in backbranches). However, it makes some sense to try to keep the latest checkpoint contained fully in a page, or at least in a single WAL segment, just on general robustness grounds. If something goes awfully wrong, it's more likely that you can recover the latest WAL segment, than the last two WAL segments. So I added an extra check that the checkpoint is not skipped if the previous checkpoint crossed a WAL segment. Reported by Jeff Janes. http://git.postgresql.org/pg/commitdiff/3d80a1e0e3e278edc6022d642478dcbd089d4483
  • Shut down test servers after pg_rewind regression tests. Now that the test servers are initialized twice in each .pl script, the single END block is not enough to stop them. Add a new clean_rewind_test function that is called at the end of each test. Michael Paquier http://git.postgresql.org/pg/commitdiff/0d8a22a9ac6a61b7993abb642cb7e4645f4087b0
  • Minor cleanup of pg_rewind. Update comments and function names to use the terms "source" and "target" consistently. Some places were calling them remote and local instead, which was confusing. Fix incorrect comment in extractPageInfo on database creation record - it was wrong on what happens for databases created in the target that don't exist in source. http://git.postgresql.org/pg/commitdiff/41457fcf970f0ec78004cc0f7b29f1d37021fbfb
  • Add missing newlines to error messages. http://git.postgresql.org/pg/commitdiff/b5e384e374657ead815a3393ca59333910611a24
  • Error out in pg_rewind if lstat() fails. A "file not found" is expected if the source server is running, so don't complain about that. But any other error is definitely not expected. http://git.postgresql.org/pg/commitdiff/b5e560c24603e5325a81055c8f36cc45d48609e4
  • Fix assertion failure in logical decoding. Logical decoding set SnapshotData's regd_count field to avoid the snapshot manager from prematurely freeing snapshots that are generated by the decoding system. That was always an abuse of the field, as it was never supposed to be used outside the snapshot manager. Commit 94028691 made snapshot manager's tracking of the snapshots smarter, and that scheme fell apart. The snapshot manager got confused and hit the assertion, when a snapshot that was marked with regd_count==1 was not found in the heap, where the snapshot manager tracks registered the snapshots. To fix, don't abuse the regd_count field like that. Logical decoding still abuses the active_count field for similar purposes, but that's currently harmless. The assertion failure was first reported by Michael Paquier http://git.postgresql.org/pg/commitdiff/e2999abcd14540e66b72deeff75662c1672d7744

Álvaro Herrera a poussé :

Bruce Momjian a poussé :

Stephen Frost a poussé :

  • Fix typo in relcache's equalPolicy(). The USING policies were not being checked for differences as the same policy was being passed in to both sides of the equal(). This could result in backends not realizing that a policy had been changed, if none of the other attributes had been changed. Fix by passing to equal() the policy1 and policy2 using quals for comparison. No need to back-patch as this is not yet released. Noticed while testing changes to RLS proposed by Dean Rasheed. http://git.postgresql.org/pg/commitdiff/ab6d1cd26ebfbfce275cd31af82814c0620e70a2

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Stephen Frost sent in another revision of a patch to use GRANT for access to privileged functions.
  • Etsuro Fujita sent in two more revisions of a patch to fix some oddities with EvalQualPlan for FDW queries involving system columns.
  • David Steele sent in three more revisions of a patch to implement pg_audit.
  • Simon Riggs sent in another revision of a patch to disable HOT cleanup, sometimes.
  • Etsuro Fujita sent in another revision of a patch to fix some infelicities in the new feature of having foreign tables in inheritance trees.
  • Etsuro Fujita sent in a patch to allow WITH OIDs when issuing CREATE FOREIGN TABLE, making it consistently able with ALTER FOREIGN TABLE.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT IGNORE (and UPDATE).
  • Zhang Zq sent in a patch to fix an issue where duplicate checkpoints were being inserted when the system was idle.
  • Qingqing Zhou sent in a patch to assert there is no duplicated exit callbacks.
  • Qingqing Zhou sent in a patch to use outerPlanState() consistently in the executor code.
  • Álvaro Herrera sent in a patch to show xl_prev in xlog.c errcontext.
  • J.L. Tallon sent in a patch to implement a new GUC (default_index_tablespace) plus supporting code.
  • Michael Paquier sent in a doc patch to warn about possible information leakage from compressing full-page writes.
  • Etsuro Fujita sent in a patch to fix some markup in config.sgml.
  • Abhijit Menon-Sen sent in another revision of a patch to recursively fsync PGDATA at startup if needed.
  • Michael Paquier sent in a patch to fix broken Install.bat when target directory contains a space.
  • Shigeru HANADA sent in two more revisions of a patch to add custom and foreign JOIN APIs.
  • Alexander Korotkov sent in another revision of a patch to implement KNN-GiST with recheck.
  • Zhang Zq sent in a patch to implement xidin.
  • Abhijit Menon-Sen sent in a patch to avoid dividing by zero when calculating percentages.
  • Andres Freund sent in another revision of a patch to iIntroduce replication identifiers.
  • Petr Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • Dmitry Dolgov sent in another revision of a patch to add some useful functions for manipulating and pretty-printing JSON.
  • Michael Paquier sent in a patch to add a missing --debug for remote mode in pg_rewind tests to RewindTest.pm.
  • Michael Paquier sent in another revision of a patch to add support for TAP tests on Windows and prefer IPC's run to system_or_bail in pg_rewind tests.
  • Mikko Tiihonen sent in a patch to libpq which allows specifying multiple host names to try to connect to.
  • Tomas Vondra sent in a patch to allow initdb-time support for various compression algorithms.

par N Bougain le samedi 25 avril 2015 à 11h47

dimanche 19 avril 2015

Guillaume Lelarge

Analyse du VACUUM

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

Et voilà, deux nouveaux chapitres écrits, dont un sur le système transactionnel de PostgreSQL. Ce dernier m'a demandé d' étudier plus attentivement le travail de l'opération VACUUM. J'en connaissais le principe et son fonctionnement, à savoir un fonctionnement en trois phases : recherche des éléments à flagguer comme invisibles, suppression de ces éléments dans les index, puis suppression dans la table (pas physiquement). Cependant, je ne l'avais pas regardé plus précisément.

J'ai donc lu le code, puis écrit un petit patch pour mieux suivre cela (disponible en pièce jointe). J'ai exécuté un script SQL pour visualiser différents comportements. Par exemple, on ajoute dix lignes dans une nouvelle table, puis on met à jour une ligne sur trois, et enfin on exécute un VACUUM sur cette table :

CREATE TABLE t2(c1 integer);
ALTER TABLE t2 SET (autovacuum_enabled=off);
INSERT INTO t2 SELECT generate_series(1, 10);
UPDATE t2 SET c1=-c1 where c1%3=1;
SET client_min_messages to log;
VACUUM t2;

Voici le log fourni par le patch :

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82231
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82231) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 1 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 4 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 11
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 12
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 7 is now REDIRECTed to item 13
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 10 is now REDIRECTed to item 14
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (10 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 320)

Il n'y a là qu'une seule étape exécutée. En effet, dû au très petit nombre de lignes dans la table, le seul bloc de 8 Ko n'a pas été entièrement occupé. Du coup, PostgreSQL place les nouvelles versions des lignes mises à jour dans le même bloc que les anciennes versions et utilise le Heap Over Tuple pour lier les enregistrements. De plus, comme il n'y a pas d'index, pas besoin de les mettre à jour.

Maintenant, faisons la même chose avec 400 lignes (en fait, suffisamment pour remplir plus d'un bloc). Les logs sont beaucoup plus importants.

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82234
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82234) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 3 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 76 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 1 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 4 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 223 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 226 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (150 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 4800)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 1 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 4 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 223 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 226 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 1 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 4 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 223 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 226 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (150 have storage, 76 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 4800)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 58 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 3 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 6 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 171 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 174 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (168 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5376)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 3 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 6 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 9 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 171 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 174 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 3 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 6 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 171 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 174 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (168 have storage, 58 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5376)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 2
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 0 deleted items found

Chaque élément devenu invisible est déclaré DEAD lors de la première étape, puis UNUSED à la troisième étape.

Il est dit que le fillfactor permet d'augmenter l'utilisation du Heap Over Tuple. Voici ce que cela donne avec un fillfactor à 90% :

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82237
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82237) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 3 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 68 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 205
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 206
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 61 is now REDIRECTed to item 225
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 64 is now REDIRECTed to item 226
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 67 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 70 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 199 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 202 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (158 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5056)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 67 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 70 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 199 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 202 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 67 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 70 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 199 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 202 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (158 have storage, 46 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5056)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 66 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 205
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 206
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 61 is now REDIRECTed to item 225
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 64 is now REDIRECTed to item 226
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 67 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 70 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 193 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 196 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (160 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5120)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 67 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 70 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 193 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 196 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 67 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 70 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 193 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 196 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (160 have storage, 44 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5120)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 2
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 0 deleted items found

Certains enregistrements bénéficient de HOT, mais la majorité deviennent UNUSED. Essayons avec un fillfactor de 50% :

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82240
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82240) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 4 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 38 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 114
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 115
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 109 is now REDIRECTed to item 150
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 112 is now REDIRECTed to item 151
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (113 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 3616)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 38 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 2 is now REDIRECTed to item 114
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 5 is now REDIRECTed to item 115
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 110 is now REDIRECTed to item 150
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 113 is now REDIRECTed to item 151
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (113 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 3616)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 2
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 37 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 3 is now REDIRECTed to item 114
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 6 is now REDIRECTed to item 115
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 108 is now REDIRECTed to item 149
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 111 is now REDIRECTed to item 150
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (113 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 3616)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 3
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 21 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 62
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 63
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 58 is now REDIRECTed to item 81
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 61 is now REDIRECTed to item 82
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (61 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 1952)

Dans ce cas, tous les enregistrements bénéficient de HOT, aucun n'est UNUSED. Cela permet des mises à jour et une maintenance plus rapides, mais c'est au prix d'une table plus volumineuse (sur disque et dans le cache des relations de PostgreSQL).

Et du coup, vous vous demandez peut-être quand va sortir la version beta 0.3 du livre ? D'ici peu a priori. Un peu de relecture, quelques ajustements de dernières minutes, et ça devrait être prêt :)

par Guillaume Lelarge le dimanche 19 avril 2015 à 17h21

mardi 14 avril 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 12 avril 2015

L'appel à conférenciers pour le PGDay de Belfort (France) se termine le 13 avril 2015. La conférence aura lieu le 2 juin : http://select-2-6-2015-as-pgday.org

La 4ème conférence PostgreSQL turque se tiendra à Istanbul le 9 mai 2015 : http://pgday.postgresql.org.tr/en/index.html

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Fujii Masao a poussé :

Álvaro Herrera a poussé :

  • Fix object identities for pg_conversion objects. This was already fixed in 0d906798f, but I failed to update the array-formatted case. This is not backpatched, since this only affects the code path introduced by commit a676201490c. http://git.postgresql.org/pg/commitdiff/70dc2db7f1dfdecdacf595bf00964cb20ad5a835
  • pg_event_trigger_dropped_objects: add is_temp column. It now also reports temporary objects dropped that are local to the backend. Previously we weren't reporting any temp objects because it was deemed unnecessary; but as it turns out, it is necessary if we want to keep close track of DDL command execution inside one session. Temp objects are reported as living in schema pg_temp, which works because such a schema-qualification always refers to the temp objects of the current session. http://git.postgresql.org/pg/commitdiff/e9a077cad3799b41e8deef6fd8cb87e50164a791
  • Remove variable shadowing. Commit a2e35b53 should have removed the variable declaration in the inner block, but didn't. As a result, the returned address might end up not being what was intended. http://git.postgresql.org/pg/commitdiff/4e17e32f53c2de4a862ee5ef8bdcfa9152c11e25
  • Fix autovacuum launcher shutdown sequence. It was previously possible to have the launcher re-execute its main loop before shutting down if some other signal was received or an error occurred after getting SIGTERM, as reported by Qingqing Zhou. While investigating, Tom Lane further noticed that if autovacuum had been disabled in the config file, it would misbehave by trying to start a new worker instead of bailing out immediately -- it would consider itself as invoked in emergency mode. Fix both problems by checking the shutdown flag in a few more places. These problems have existed since autovacuum was introduced, so backpatch all the way back. http://git.postgresql.org/pg/commitdiff/5df64f298d2863c9fb39437abb3ae6f988aedc0a
  • Change SQLSTATE for event triggers "wrong context" message. When certain event-trigger-only functions are called when not in the wrong context, they were reporting the "feature not supported" SQLSTATE, which is somewhat misleading. Create a new custom error code for such uses instead. Not backpatched since it may be seen as an undesirable behavioral change. Author: Michael Paquier. Discussion: https://www.postgresql.org/message-id/CAB7nPqQ-5NAkHQHh_NOm7FPep37NCiLKwPoJ2Yxb8TDoGgbYYA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/73206812cd97436cffd8f331dbb09d38a2728162
  • Optimize locking a tuple already locked by another subxact. Locking and updating the same tuple repeatedly led to some strange multixacts being created which had several subtransactions of the same parent transaction holding locks of the same strength. However, once a subxact of the current transaction holds a lock of a given strength, it's not necessary to acquire the same lock again. This made some coding patterns much slower than required. The fix is twofold. First we change HeapTupleSatisfiesUpdate to return HeapTupleBeingUpdated for the case where the current transaction is already a single-xid locker for the given tuple; it used to return HeapTupleMayBeUpdated for that case. The new logic is simpler, and the change to pgrowlocks is a testament to that: previously we needed to check for the single-xid locker separately in a very ugly way. That test is simpler now. As fallout from the HTSU change, some of its callers need to be amended so that tuple-locked-by-own-transaction is taken into account in the BeingUpdated case rather than the MayBeUpdated case. For many of them there is no difference; but heap_delete() and heap_update now check explicitely and do not grab tuple lock in that case. The HTSU change also means that routine MultiXactHasRunningRemoteMembers introduced in commit 11ac4c73cb895 is no longer necessary and can be removed; the case that used to require it is now handled naturally as result of the changes to heap_delete and heap_update. The second part of the fix to the performance issue is to adjust heap_lock_tuple to avoid the slowness: 1. Previously we checked for the case that our own transaction already held a strong enough lock and returned MayBeUpdated, but only in the multixact case. Now we do it for the plain Xid case as well, which saves having to LockTuple. 2. If the current transaction is the only locker of the tuple (but with a lock not as strong as what we need; otherwise it would have been caught in the check mentioned above), we can skip sleeping on the multixact, and instead go straight to create an updated multixact with the additional lock strength. 3. Most importantly, make sure that both the single-xid-locker case and the multixact-locker case optimization are applied always. We do this by checking both in a single place, rather than them appearing in two separate portions of the routine -- something that is made possible by the HeapTupleSatisfiesUpdate API change. Previously we would only check for the single-xid case when HTSU returned MayBeUpdated, and only checked for the multixact case when HTSU returned BeingUpdated. This was at odds with what HTSU actually returned in one case: if our own transaction was locker in a multixact, it returned MayBeUpdated, so the optimization never applied. This is what led to the large multixacts in the first place. Per bug report #8470 by Oskari Saarenmaa. http://git.postgresql.org/pg/commitdiff/27846f02c176eebe7e08ce51ed4d52140454e196

Simon Riggs a poussé :

Heikki Linnakangas a poussé :

Tom Lane a poussé :

Robert Haas a poussé :

Bruce Momjian a poussé :

Andres Freund a poussé :

Magnus Hagander a poussé :

Peter Eisentraut a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Fabrízio de Royes Mello sent in two more revisions of a patch to refactor reloptions to set locklevel.
  • Artem Luzyanin sent in two revisions of a patch to consolidate the documentation of spinlocks and like kind items.
  • Emre Hasegeli sent in another revision of a patch to add a BRIN range operator class.
  • Tomas Vondra sent in a patch to use foreign keys to improve join estimates.
  • Michael Paquier sent in a patch to ignore some binaries generated in src/test.
  • SAWADA Masahiko and Fabrízio de Royes Mello traded patches to add REINDEX ... VERBOSE.
  • Shigeru HANADA sent in four more revisions of a patch to add a foreign join API.
  • Petr Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • Craig Ringer sent in a patch to add a pid column to pg_replication_slots.
  • Peter Eisentraut and Pavel Stehule traded patches to add TRANSFORMS.
  • Michael Paquier sent in another revision of a patch to add an error code to track unsupported contexts.
  • Álvaro Herrera sent in two more revisions of a patch to add deparsing utilities.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT UPDATE (and IGNORE).
  • Tom Lane sent in another revision of a patch to implement UPDATE (*) SET ...
  • Fujii Masao sent in another revision of a patch to remove the obsolete FORCE option from REINDEX.
  • Dean Rasheed sent in another revision of a patch to fix some infelicities in the error reporting for row-level access control.
  • Kyotaro HORIGUCHI sent in another revision of a patch to implement regnamespace and regrole.
  • Antonin Houska sent in a patch to fix some issues in xlogreader.
  • Craig Ringer sent in two revisions of a patch to make pg_dump -t take materialized views, matview data, foreign tables, and sequences.
  • Bruce Momjian sent in two revisions of a patch to ensure that CREATE TABLE (LIKE...) preserves the relhasoids setting.
  • Michael Paquier sent in another revision of a patch to suppport TAP tests with MSVC and Windows.
  • Jan Urbański sent in another revision of a patch to fix a bug in libpq's multi-threaded SSL callback handling.
  • Ian Stakenvicius sent in a patch to fix an issue on Gentoo where postgres fails to start with timezone-data >=2013e.
  • Etsuro Fujita sent in another revision of a patch to fix a problem where EvalPlanQual behaves oddly for FDW queries involving system columns.
  • Pavel Stehule sent in a patch to add a "raw" output option to COPY.
  • Chen Huajun sent in a patch to prevent setting Win32 server-side socket buffer size on Windows 2012.
  • David Rowley sent in a patch to fix a few appendStringInfo* calls that were not quite doing things the way as intended.
  • Michael Paquier sent in another revision of a patch to help improve the performance of make check-world.
  • Heikki Linnakangas sent in a patch to remove xlogrecord padding.

par N Bougain le mardi 14 avril 2015 à 21h05

samedi 11 avril 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 5 avril 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Fix multiple bugs and infelicities in pg_rewind. Bugs all spotted by Coverity, including wrong realloc() size request and memory leaks. Cosmetic improvements by me. The usage of the global variable "filemap" here is still pretty awful, but at least I got rid of the gratuitous aliasing in several routines (which was helping to annoy Coverity, as well as being a bug risk). http://git.postgresql.org/pg/commitdiff/c67f366fa9f748257861ee233b47b80eb5ffa857
  • Clean up all the cruft after a pg_rewind test run. regress_log temp directory was properly .gitignore'd, which may explain why it got left out of the "make clean" action. http://git.postgresql.org/pg/commitdiff/1c41e2a998a0de16d9d33949a7b98a5be3d2477c
  • Fix rare core dump in BackendIdGetTransactionIds(). BackendIdGetTransactionIds() neglected the possibility that the PROC pointer in a ProcState array entry is null. In current usage, this could only crash if the other backend had exited since pgstat_read_current_status saw it as active, which is a pretty narrow window. But it's reachable in the field, per bug #12918 from Vladimir Borodin. Back-patch to 9.4 where the faulty code was introduced. http://git.postgresql.org/pg/commitdiff/701dcc983eb4d08dd36bb3a0ddba255819797760
  • Be more careful about printing constants in ruleutils.c. The previous coding in get_const_expr() tried to avoid quoting integer, float, and numeric literals if at all possible. While that looks nice, it means that dumped expressions might re-parse to something that's semantically equivalent but not the exact same parsetree; for example a FLOAT8 constant would re-parse as a NUMERIC constant with a cast to FLOAT8. Though the result would be the same after constant-folding, this is problematic in certain contexts. In particular, Jeff Davis pointed out that this could cause unexpected failures in ALTER INHERIT operations because of child tables having not-exactly-equivalent CHECK expressions. Therefore, favor correctness over legibility and dump such constants in quotes except in the limited cases where they'll be interpreted as the same type even without any casting. This results in assorted small changes in the regression test outputs, and will affect display of user-defined views and rules similarly. The odds of that causing problems in the field seem non-negligible; given the lack of previous complaints, it seems best not to change this in the back branches. http://git.postgresql.org/pg/commitdiff/542320c2bd0b3796a8a9a4617cdb23fbad473390
  • Fix bogus concurrent use of _hash_getnewbuf() in bucket split code. _hash_splitbucket() obtained the base page of the new bucket by calling _hash_getnewbuf(), but it held no exclusive lock that would prevent some other process from calling _hash_getnewbuf() at the same time. This is contrary to _hash_getnewbuf()'s API spec and could in fact cause failures. In practice, we must only call that function while holding write lock on the hash index's metapage. An additional problem was that we'd already modified the metapage's bucket mapping data, meaning that failure to extend the index would leave us with a corrupt index. Fix both issues by moving the _hash_getnewbuf() call to just before we modify the metapage in _hash_expandtable(). Unfortunately there's still a large problem here, which is that we could also incur ENOSPC while trying to get an overflow page for the new bucket. That would leave the index corrupt in a more subtle way, namely that some index tuples that should be in the new bucket might still be in the old one. Fixing that seems substantially more difficult; even preallocating as many pages as we could possibly need wouldn't entirely guarantee that the bucket split would complete successfully. So for today let's just deal with the base case. Per report from Antonin Houska. Back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/ed9cc2b5df59fdbc50cce37399e26b03ab2c1686
  • Fix incorrect markup in documentation of window frame clauses. You're required to write either RANGE or ROWS to start a frame clause, but the documentation incorrectly implied this is optional. Noted by David Johnston. http://git.postgresql.org/pg/commitdiff/f6caf5acf1def92d7425151a92fd990c566fdcc3
  • Provide real selectivity estimators for inet/cidr operators. This patch fills in the formerly-stub networksel() and networkjoinsel() estimation functions. Those are used for << <<= >> >>= and && operators on inet/cidr types. The estimation is not perfect, certainly, because we rely on the existing statistics collected for the inet btree operators. But it's a long way better than nothing, and it's not clear that asking ANALYZE to collect separate stats for these operators would be a win. Emre Hasegeli, with reviews from Dilip Kumar and Heikki Linnakangas, and some further hacking by me http://git.postgresql.org/pg/commitdiff/89840d7d3fa943cb932f6a00707fdb17a9cab001
  • Fix rare startup failure induced by MVCC-catalog-scans patch. While a new backend nominally participates in sinval signaling starting from the SharedInvalBackendInit call near the top of InitPostgres, it cannot recognize sinval messages for unshared catalogs of its database until it has set up MyDatabaseId. This is not problematic for the catcache or relcache, which by definition won't have loaded any data from or about such catalogs before that point. However, commit 568d4138c646cd7c introduced a mechanism for re-using MVCC snapshots for catalog scans, and made invalidation of those depend on recognizing relevant sinval messages. So it's possible to establish a catalog snapshot to read pg_authid and pg_database, then before we set MyDatabaseId, receive sinval messages that should result in invalidating that snapshot --- but do not, because we don't realize they are for our database. This mechanism explains the intermittent buildfarm failures we've seen since commit 31eae6028eca4365. That commit was not itself at fault, but it introduced a new regression test that does reconnections concurrently with the "vacuum full pg_am" command in vacuum.sql. This allowed the pre-existing error to be exposed, given just the right timing, because we'd fail to update our information about how to access pg_am. In principle any VACUUM FULL on a system catalog could have created a similar hazard for concurrent incoming connections. Perhaps there are more subtle failure cases as well. To fix, force invalidation of the catalog snapshot as soon as we've set MyDatabaseId. Back-patch to 9.4 where the error was introduced. http://git.postgresql.org/pg/commitdiff/bc49d9324a464fce8f60e1bc14531631883021d4
  • Remove unnecessary variables in _hash_splitbucket(). Commit ed9cc2b5df59fdbc50cce37399e26b03ab2c1686 made it unnecessary to pass start_nblkno to _hash_splitbucket(), and for that matter unnecessary to have the internal nblkno variable either. My compiler didn't complain about that, but some did. I also rearranged the use of oblkno a bit to make that case more parallel. Report and initial patch by Petr Jelinek, rearranged a bit by me. Back-patch to all branches, like the previous patch. http://git.postgresql.org/pg/commitdiff/b7e1652d5de8b618c0204588969c8b59d12e9361
  • Fix TAP tests to use only standard command-line argument ordering. Some of the TAP tests were supposing that PG programs would accept switches after non-switch arguments on their command lines. While GNU getopt_long() does allow that, our own implementation does not, and it's nowhere suggested in our documentation that such cases should work. Adjust the tests to use only the documented syntax. Back-patch to 9.4, since without this the TAP tests fail when run with src/port's getopt_long() implementation. Michael Paquier http://git.postgresql.org/pg/commitdiff/c67a86f7da90c30b81f91957023fb752f06f0598
  • Fix incorrect matching of subexpressions in outer-join plan nodes. Previously we would re-use input subexpressions in all expression trees attached to a Join plan node. However, if it's an outer join and the subexpression appears in the nullable-side input, this is potentially incorrect for apparently-matching subexpressions that came from above the outer join (ie, targetlist and qpqual expressions), because the executor will treat the subexpression value as NULL when maybe it should not be. The case is fairly hard to hit because (a) you need a non-strict subexpression (else NULL is correct), and (b) we don't usually compute expressions in the outputs of non-toplevel plan nodes. But we might do so if the expressions are sort keys for a mergejoin, for example. Probably in the long run we should make a more explicit distinction between Vars appearing above and below an outer join, but that will be a major planner redesign and not at all back-patchable. For the moment, just hack set_join_references so that it will not match any non-Var expressions coming from nullable inputs to expressions that came from above the join. (This is somewhat overkill, in that a strict expression could still be matched, but it doesn't seem worth the effort to check that.) Per report from Qingqing Zhou. The added regression test case is based on his example. This has been broken for a very long time, so back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/ca6805338fba010cc3f8b842905d7a62e280b7ab
  • Suppress clang's unhelpful gripes about -pthread switch being unused. Considering the number of cases in which "unused" command line arguments are silently ignored by compilers, it's fairly astonishing that anybody thought this warning was useful; it's certainly nothing but an annoyance when building Postgres. One such case is that neither gcc nor clang complain about unrecognized -Wno-foo switches, making it more difficult to figure out whether the switch does anything than one could wish. Back-patch to 9.3, which is as far back as the patch applies conveniently (we'd have to back-patch PGAC_PROG_CC_VAR_OPT to go further, and it doesn't seem worth that). http://git.postgresql.org/pg/commitdiff/73b416b2e41237b657d29d8f42a4bb34bf700928

Heikki Linnakangas a poussé :

Álvaro Herrera a poussé :

  • Fix lost persistence setting during REINDEX INDEX. ReindexIndex() trusts a parser-built RangeVar with the persistence to use for the new copy of the index; but the parser naturally does not know what's the persistence of the original index. To find out the correct persistence, grab it from relcache. This bug was introduced by commit 85b506bbfc2937c9, and therefore no backpatch is necessary. Bug reported by Thom Brown, analysis and patch by Michael Paquier; test case provided by Fabrízio de Royes Mello. http://git.postgresql.org/pg/commitdiff/0853630159944bb3652336602ff5f7f62cd27a5a
  • Change array_offset to return subscripts, not offsets ... and rename it and its sibling array_offsets to array_position and array_positions, to account for the changed behavior. Having the functions return subscripts better matches existing practice, and is better suited to using the result value as a subscript into the array directly. For one-based arrays, the new definition is identical to what was originally committed. (We use the term "subscript" in the documentation, which is what we use whenever we talk about arrays; but the functions themselves are named using the word "position" to match the standard-defined POSITION() functions.) Author: Pavel Stěhule. Behavioral problem noted by Dean Rasheed. http://git.postgresql.org/pg/commitdiff/97690ea6e86c412461dd5dc99953b829564d1a55
  • psql: fix \connect with URIs and conninfo strings. psql was already accepting conninfo strings as the first parameter in \connect, but the way it worked wasn't sane; some of the other parameters would get the previous connection's values, causing it to connect to a completely unexpected server or, more likely, not finding any server at all because of completely wrong combinations of parameters. Fix by explicitely checking for a conninfo-looking parameter in the dbname position; if one is found, use its complete specification rather than mix with the other arguments. Also, change tab-completion to not try to complete conninfo/URI-looking "dbnames" and document that conninfos are accepted as first argument. There was a weak consensus to backpatch this, because while the behavior of using the dbname as a conninfo is nowhere documented for \connect, it is reasonable to expect that it works because it does work in many other contexts. Therefore this is backpatched all the way back to 9.0. To implement this, routines previously private to libpq have been duplicated so that psql can decide what looks like a conninfo/URI string. In back branches, just duplicate the same code all the way back to 9.2, where URIs where introduced; 9.0 and 9.1 have a simpler version. In master, the routines are moved to src/common and renamed. Author: David Fetter, Andrew Dunstan. Some editorialization by me (probably earning a Gierth's "Sloppy" badge in the process.) Reviewers: Andrew Gierth, Erik Rijkers, Pavel Stěhule, Stephen Frost, Robert Haas, Andrew Dunstan. http://git.postgresql.org/pg/commitdiff/fcef1617295c074f2684c887627184d2fc26ac04
  • psql: fix \connect with URIs and conninfo strings. This is the second try at this, after fcef1617295 failed miserably and had to be reverted: as it turns out, libpq cannot depend on libpgcommon after all. Instead of shuffling code in the master branch, make that one just like 9.4 and accept the duplication. (This was all my own mistake, not the patch submitter's). psql was already accepting conninfo strings as the first parameter in \connect, but the way it worked wasn't sane; some of the other parameters would get the previous connection's values, causing it to connect to a completely unexpected server or, more likely, not finding any server at all because of completely wrong combinations of parameters. Fix by explicitely checking for a conninfo-looking parameter in the dbname position; if one is found, use its complete specification rather than mix with the other arguments. Also, change tab-completion to not try to complete conninfo/URI-looking "dbnames" and document that conninfos are accepted as first argument. There was a weak consensus to backpatch this, because while the behavior of using the dbname as a conninfo is nowhere documented for \connect, it is reasonable to expect that it works because it does work in many other contexts. Therefore this is backpatched all the way back to 9.0. Author: David Fetter, Andrew Dunstan. Some editorialization by me (probably earning a Gierth's "Sloppy" badge in the process.) Reviewers: Andrew Gierth, Erik Rijkers, Pavel Stěhule, Stephen Frost, Robert Haas, Andrew Dunstan. http://git.postgresql.org/pg/commitdiff/e146ca682062ca1f5015f3820571c5359f5f9dba
  • autovacuum: Fix polarity of "wraparound" variable. Commit 0d831389749a3 inadvertently reversed the meaning of the wraparound variable. This causes vacuums which are not required for wraparound to wait for locks to be acquired, and what is worse, it allows wraparound vacuums to skip locked pages. Bug reported by Jeff Janes in http://www.postgresql.org/message-id/CAMkU=1xmTEiaY=5oMHsSQo5vd9V1Ze4kNLL0qN2eH0P_GXOaYw@mail.gmail.com Analysis and patch by Kyotaro HORIGUCHI http://git.postgresql.org/pg/commitdiff/00ee6c7672fe0bf9448bc744b5e3408f5ebffc2e
  • Have autovacuum workers listen to SIGHUP, too. They have historically ignored it, but it's been said to be useful at times to change their settings mid-flight. Author: Michael Paquier http://git.postgresql.org/pg/commitdiff/a75fb9b335db0e063ece283ebd207530abe1b53b
  • Add log_min_autovacuum_duration per-table option. This is useful to control autovacuum log volume, for situations where monitoring only a set of tables is necessary. Author: Michael Paquier Reviewed by: A team led by Naoya Anzai (also including Akira Kurosawa, Taiki Kondo, Huong Dangminh), Fujii Masao. http://git.postgresql.org/pg/commitdiff/4ff695b17d32a9c330952192dbc789d31a5e2f5e
  • Transform ALTER TABLE/SET TYPE/USING expr during parse analysis. This lets later stages have access to the transformed expression; in particular it allows DDL-deparsing code during event triggers to pass the transformed expression to ruleutils.c, so that the complete command can be deparsed. This shuffles the timing of the transform calls a bit: previously, nothing was transformed during parse analysis, and only the RELKIND_RELATION case was being handled during execution. After this patch, all expressions are transformed during parse analysis (including those for relkinds other than RELATION), and the error for other relation kinds is thrown only during execution. So we do more work than before to reject some bogus cases. That seems acceptable. http://git.postgresql.org/pg/commitdiff/9550e8348b7965715789089555bb5a3fda8c269c

Andrew Dunstan a poussé :

  • Run pg_upgrade and pg_resetxlog with restricted token on Windows. As with initdb these programs need to run with a restricted token, and if they don't pg_upgrade will fail when run as a user with Adminstrator privileges. Backpatch to all live branches. On the development branch the code is reorganized so that the restricted token code is now in a single location. On the stable bramches a less invasive change is made by simply copying the relevant code to pg_upgrade.c and pg_resetxlog.c. Patches and bug report from Muhammad Asif Naeem, reviewed by Michael Paquier, slightly edited by me. http://git.postgresql.org/pg/commitdiff/fa1e5afa8a26d467aec7c8b36a0b749b690f636c
  • Enable float8-byval as the default for 64 bit MSVC builds. This is a long-standing inconsistency that was probably just missed when we got 64 bit MSVC builds. This brings the platform into line with all other systems. http://git.postgresql.org/pg/commitdiff/cf376a4adc0805b0960a5f8e8325fae7d4456926

Bruce Momjian a poussé :

Fujii Masao a poussé :

Simon Riggs a poussé :

Robert Haas a poussé :

Andres Freund a poussé :

  • Define integer limits independently from the system definitions. In 83ff1618 we defined integer limits iff they're not provided by the system. That turns out not to be the greatest idea because there's different ways some datatypes can be represented. E.g. on OSX PG's 64bit datatype will be a 'long int', but OSX unconditionally uses 'long long'. That disparity then can lead to warnings, e.g. around printf formats. One way to fix that would be to back int64 using stdint.h's int64_t. While a good idea it's not that easy to implement. We would e.g. need to include stdint.h in our external headers, which we don't today. Also computing the correct int64 printf formats in that case is nontrivial. Instead simply prefix the integer limits with PG_ and define them unconditionally. I've adjusted all the references to them in code, but not the ones in comments; the latter seems unnecessary to me. Discussion: 20150331141423.GK4878@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/62e2a8dc2c7f6b1351a0385491933af969ed4265

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in a doc patch to describe more precisely the rounding behavior of numeric and double precision.
  • Tomas Vondra sent in another revision of a patch to implement multivariate statistics, useful for among other things cross-column correlations.
  • Heikki Linnakangas sent in a patch to implement SCRAM authentication.
  • David Fetter sent in two more revisions of a patch to allow make_date() to use negative years for BCE dates.
  • Craig Ringer sent in a patch to add views, materialized views, and foreign tables to the meaning of pg_dump's -t command line argument.
  • Fabrízio de Royes Mello sent in a WIP patch to reduce lock level when setting autovacuum reloptions in "ALTER TABLE .. SET ( .. )" statement.
  • Pavel Stehule sent in another revision of a patch to add a row_to_array() function.
  • Michael Paquier sent in two revisions of a patch to add some tests for pg_rewind.
  • Abhijit Menon-Sen sent in two more revisions of a patch to add pgstatbloat to pgstattuple.
  • Haribabu Kommi sent in another revision of a patch to add a catalog view to for pg_hba.conf.
  • Tomas Vondra sent in another revision of a patch to improve the n-distinct estimator.
  • Jehan-Guillaume de Rorthais sent in a patch to document the maximum number of files in the pg_xlog directory.
  • Stephen Frost sent in another revision of a patch to implement role attributes.
  • Bruce Momjian sent in two more revisions of a patch to fix some infelicities in zero-padding in to_char().
  • David Steele sent in another revision of a patch to implement pg_audit.
  • Bruce Momjian sent in two revisions of a patch to fix an infelicity in the SSPI error reporting code.
  • Heikki Linnakangas sent in another revision of a patch to use Intel SSE4.2 CRC instructions where available.
  • Petr Jelinek sent in a patch to remove an unused variable in src/backend/access/hash/hashpage.c.
  • Petr Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • SAWADA Masahiko sent in a WIP patch to allow making read-only tables.
  • Shigeru HANADA sent in another revision of a patch to add join push-down support to postgres_fdw.
  • Etsuro Fujita sent in another revision of a patch to fix some infelicities in the way EvalPlanQual behaves for FDW queries involving system columns.
  • SAWADA Masahiko sent in another revision of a patch to create a pg_file_settings view.
  • Tomas Vondra sent in a patch to fix to get rid of the spurious warnings caused by the recent change to _hash_splitbucket().
  • Peter Geoghegan sent in a patch to make trace_sort control abbreviation debug output for the text opclass, making it consistent with the numeric opclass.

par N Bougain le samedi 11 avril 2015 à 16h46

Nouvelles hebdomadaires de PostgreSQL - 29 mars 2015

Le pgDay Paris 2015 aura lieu le 21 avril à Paris au siège de La Poste. Les inscriptions sont ouvertes : http://pgday.paris/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mars

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Robert Haas a poussé :

Andres Freund a poussé :

  • Fix copy & paste error in 4f1b890b137. Due to the bug delayed standbys would not delay when applying prepared transactions. Discussion: CAB7nPqT6BO1cCn+sAyDByBxA4EKZNAiPi2mFJ=ANeZmnmewRyg@mail.gmail.com Michael Paquier via Coverity. http://git.postgresql.org/pg/commitdiff/a1105c3dd44c1fb76eb62a708f0421f21b9dde9b
  • Don't delay replication for less than recovery_min_apply_delay's resolution. Recovery delays are implemented by waiting on a latch, and latches take milliseconds as a parameter. The required amount of waiting was computed using microsecond resolution though and the wait loop's abort condition was checking the delay in microseconds as well. This could lead to short spurts of busy looping when the overall wait time was below a millisecond, but above 0 microseconds. Instead just formulate the wait loop's abort condition in millisecond granularity as well. Given that that's recovery_min_apply_delay resolution, it seems harmless to not wait for less than a millisecond. Backpatch to 9.4 where recovery_min_apply_delay was introduced. Discussion: 20150323141819.GH26995@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/87cec51d3ad1107f6f224ed7d773e70c8896e4c0
  • Centralize definition of integer limits. Several submitted and even committed patches have run into the problem that C89, our baseline, does not provide minimum/maximum values for various integer datatypes. C99's stdint.h does, but we can't rely on it. Several parts of the code defined limits locally, so instead centralize the definitions to c.h. This patch also changes the more obvious usages of literal limit values; there's more places that could be changed, but it's less clear whether it's beneficial to change those. Author: Andrew Gierth Discussion: 87619tc5wc.fsf@news-spur.riddles.org.uk http://git.postgresql.org/pg/commitdiff/83ff1618bc9d4e530d3ef2a668a71326784a753c

Heikki Linnakangas a poussé :

Álvaro Herrera a poussé :

  • vacuumdb: Check result status of PQsendQuery. Noticed by Coverity http://git.postgresql.org/pg/commitdiff/871293fb7fa869fbe0f202bef2e0d781bfd21b3e
  • Fix gram.y comment to match reality. There are other comments in there that don't precisely match what's implemented, but this one confused me enough to be worth fixing. http://git.postgresql.org/pg/commitdiff/dc8e05295ab126bc4c943cab3e8e117489ecb246
  • Fix bug for array-formatted identities of user mappings. I failed to realize that server names reported in the object args array would get quoted, which is wrong; remove that, making sure that it's only quoted in the string-formatted identity. This bug was introduced by my commit cf34e373, which was backpatched, but since object name/args arrays are new in commit a676201490c8, there is no need to backpatch this any further. http://git.postgresql.org/pg/commitdiff/b3196e65f5bfc997ec7fa3f91645a09289c10dee
  • Add OID output argument to DefineTSConfiguration ... which is set to the OID of a copied text search config, whenever the COPY clause is used. This is in the spirit of commit a2e35b53c39. http://git.postgresql.org/pg/commitdiff/8217fb1441ce4b4e1785f9acfa0ce50039247a10
  • Return ObjectAddress in many ALTER TABLE sub-routines. Since commit a2e35b53c39b2a, most CREATE and ALTER commands return the ObjectAddress of the affected object. This is useful for event triggers to try to figure out exactly what happened. This patch extends this idea a bit further to cover ALTER TABLE as well: an auxiliary ObjectAddress is returned for each of several subcommands of ALTER TABLE. This makes it possible to decode with precision what happened during execution of any ALTER TABLE command; for instance, which constraint was added by ALTER TABLE ADD CONSTRAINT, or which parent got dropped from the parents list by ALTER TABLE NO INHERIT. As with the previous patch, there is no immediate user-visible change here. This is all really just continuing what c504513f83a9ee8 started. Reviewed by Stephen Frost. http://git.postgresql.org/pg/commitdiff/bdc3d7fa2376a7a1e977383cc3221cfe44c4a893

Tom Lane a poussé :

  • Apply table and domain CHECK constraints in name order. Previously, CHECK constraints of the same scope were checked in whatever order they happened to be read from pg_constraint. (Usually, but not reliably, this would be creation order for domain constraints and reverse creation order for table constraints, because of differing implementation details.) Nondeterministic results of this sort are problematic at least for testing purposes, and in discussion it was agreed to be a violation of the principle of least astonishment. Therefore, borrow the principle already established for triggers, and apply such checks in name order (using strcmp() sort rules). This lets users control the check order if they have a mind to. Domain CHECK constraints still follow the rule of checking lower nested domains' constraints first; the name sort only applies to multiple constraints attached to the same domain. In passing, I failed to resist the temptation to wordsmith a bit in create_domain.sgml. Apply to HEAD only, since this could result in a behavioral change in existing applications, and the potential regression test failures have not actually been observed in our buildfarm. http://git.postgresql.org/pg/commitdiff/e5f455f59fed0632371cddacddd79895b148dc07
  • Fix ExecOpenScanRelation to take a lock on a ROW_MARK_COPY relation. ExecOpenScanRelation assumed that any relation listed in the ExecRowMark list has been locked by InitPlan; but this is not true if the rel's markType is ROW_MARK_COPY, which is possible if it's a foreign table. In most (possibly all) cases, failure to acquire a lock here isn't really problematic because the parser, planner, or plancache would have taken the appropriate lock already. In principle though it might leave us vulnerable to working with a relation that we hold no lock on, and in any case if the executor isn't depending on previously-taken locks otherwise then it should not do so for ROW_MARK_COPY relations. Noted by Etsuro Fujita. Back-patch to all active versions, since the inconsistency has been there a long time. (It's almost certainly irrelevant in 9.0, since that predates foreign tables, but the code's still wrong on its own terms.) http://git.postgresql.org/pg/commitdiff/feeb526cfe33657b8aa8b0cdd45b2ef0d9898877
  • Upgrade src/port/rint.c to be POSIX-compliant. The POSIX spec says that rint() rounds halfway cases to nearest even. Our substitute implementation failed to do that, rather rounding halfway cases away from zero; and it also got some other cases (such as minus zero) wrong. This led to observable cross-platform differences, as reported in bug #12885 from Rich Schaaf; in particular, casting from float to int didn't honor round-to-nearest-even on builds using rint.c. Implement something that attempts to cover all cases per spec, and add some simple regression tests so that we'll notice if any platforms still get this wrong. Although this is a bug fix, no back-patch, as a behavioral change in the back branches was agreed not to be a good idea. Pedro Gimeno Fortea, reviewed by Michael Paquier and myself http://git.postgresql.org/pg/commitdiff/06bf0dd6e354c765403d1331cc9896b360754521
  • Add an ASSERT statement in plpgsql. This is meant to make it easier to insert simple debugging cross-checks in plpgsql functions. Pavel Stehule, reviewed by Jim Nasby http://git.postgresql.org/pg/commitdiff/a4847fc3ef139ba9a8ffebb6ffa06ee72078ffa2
  • Suppress some unused-variable complaints in new LOCK_DEBUG code. Jeff Janes http://git.postgresql.org/pg/commitdiff/bed756a820a2c1ee359f5f5b44806e3599190e95
  • Tweak __attribute__-wrapping macros for better pgindent results. This improves on commit bbfd7edae5aa5ad5553d3c7e102f2e450d4380d4 by making two simple changes: * pg_attribute_noreturn now takes parentheses, ie pg_attribute_noreturn(). Likewise pg_attribute_unused(), pg_attribute_packed(). This reduces pgindent's tendency to misformat declarations involving them. * attributes are now always attached to function declarations, not definitions. Previously some places were taking creative shortcuts, which were not merely candidates for bad misformatting by pgindent but often were outright wrong anyway. (It does little good to put a noreturn annotation where callers can't see it.) In any case, if we would like to believe that these macros can be used with non-gcc compilers, we should avoid gratuitous variance in usage patterns. I also went through and manually improved the formatting of a lot of declarations, and got rid of excessively repetitive (and now obsolete anyway) comments informing the reader what pg_attribute_printf is for. http://git.postgresql.org/pg/commitdiff/785941cdc359c6e595201ffb0df9d28f3f7173a4
  • Better fix for misuse of Float8GetDatumFast(). We can use that macro as long as we put the value into a local variable. Commit 735cd6128 was not wrong on its own terms, but I think this way looks nicer, and it should save a few cycles on 32-bit machines. http://git.postgresql.org/pg/commitdiff/2c33e0fbceb01d0ecd78330feef1315682c64bc4
  • Minor code cleanups in pgbench expression support. Get rid of unnecessary expr_yylex declaration (we haven't supported flex 2.5.4 in a long time, and even if we still did, the declaration in pgbench.h makes this one unnecessary and inappropriate). Fix copyright dates, improve some layout choices, etc. http://git.postgresql.org/pg/commitdiff/e9dd03c03aeed6486129e3101695b13d469c2985
  • Remove a couple other vestigial yylex() declarations. These were workarounds for a long-gone flex bug; all supported versions of flex emit an extern declaration as expected. http://git.postgresql.org/pg/commitdiff/9a8e23311cac14168df6644e03d533a4b07f933e
  • Make ginbuild's funcCtx be independent of its tmpCtx. Previously the funcCtx was a child of the tmpCtx, but that was broken by commit eaa5808e8ec4e82ce1a87103a6b6f687666e4e4c, which made MemoryContextReset() delete, not reset, child contexts. The behavior of having a tmpCtx reset also clear the other context seems rather dubious anyway, so let's just disentangle them. Per report from Erik Rijkers. In passing, fix badly-inaccurate comments about these contexts. http://git.postgresql.org/pg/commitdiff/1601830ec20d56dc7bf6b60a34f69841429e4825
  • Add vacuum_delay_point call in compute_index_stats's per-sample-row loop. Slow functions in index expressions might cause this loop to take long enough to make it worth being cancellable. Probably it would be enough to call CHECK_FOR_INTERRUPTS here, but for consistency with other per-sample-row loops in this file, let's use vacuum_delay_point. Report and patch by Jeff Janes. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/e4cbfd673d530a3e841db26a74f22e11a991205a

Bruce Momjian a poussé :

Kevin Grittner a poussé :

  • Reduce pinning and buffer content locking for btree scans. Even though the main benefit of the Lehman and Yao algorithm for btrees is that no locks need be held between page reads in an index search, we were holding a buffer pin on each leaf page after it was read until we were ready to read the next one. The reason was so that we could treat this as a weak lock to create an "interlock" with vacuum's deletion of heap line pointers, even though our README file pointed out that this was not necessary for a scan using an MVCC snapshot. The main goal of this patch is to reduce the blocking of vacuum processes by in-progress btree index scans (including a cursor which is idle), but the code rearrangement also allows for one less buffer content lock to be taken when a forward scan steps from one page to the next, which results in a small but consistent performance improvement in many workloads. This patch leaves behavior unchanged for some cases, which can be addressed separately so that each case can be evaluated on its own merits. These unchanged cases are when a scan uses a non-MVCC snapshot, an index-only scan, and a scan of a btree index for which modifications are not WAL-logged. If later patches allow all of these cases to drop the buffer pin after reading a leaf page, then the btree vacuum process can be simplified; it will no longer need the "super-exclusive" lock to delete tuples from a page. Reviewed by Heikki Linnakangas and Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/2ed5b87f96d473962ec5230fd820abfeaccb2069

Tatsuo Ishii a poussé :

Andrew Dunstan a poussé :

Peter Eisentraut a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Pavel Stehule sent in two more revisions of a patch to add a --strict option to pg_dump for missing tables.
  • Bruce Momjian sent in three more revisions of a patch to add the asciidoc output format to psql.
  • David Steele sent in two more revisions of a patch to to add a pg_audit extension.
  • Peter Geoghegan and Andrew (RhodiumToad) Gierth sent in two more revisions of a patch to add abbreviated keys for NUMERIC.
  • Michael Paquier sent in two revisions of a patch to expose PG_VERSION_NUM to developers.
  • Michael Paquier sent in another revision of a patch to add a table-level log_autovacuum_min_duration.
  • Kyotaro HORIGUCHI sent in a patch to remove individual phases to mark unique joins.
  • Andres Freund sent in another revision of a patch to add replication identifiers.
  • Amit Kapila sent in a patch to fix a bug in dsm_attach().
  • Peter Geoghegan sent in a patch to add DatumGetUInt32() around the hash_any() and hash_uint32() calls within varlena.c.
  • Heikki Linnakangas sent in another revision of a patch to use Intel SSE4.2 CRC instructions where available.
  • Álvaro Herrera sent in another revision of a patch to implement parsing utility commands.
  • Kaigai Kouhei sent in another revision of a patch to add a custom JOIN API.
  • Fabrízio de Royes Mello sent in a patch to fix a bug with indexes on unlogged tables.
  • Amit Kapila sent in two more revisions of a patch to implement parallel sequential scan.
  • Antonin Houska sent in two revisions of a WIP patch to enable spliting hash index buckets.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT IGNORE (and UPDATE).
  • Haribabu Kommi sent in another revision of a patch to add a catalog view to pg_hba.conf.
  • David Rowley sent in another revision of a patch to improve performance for outer joins where the outer side is unique.
  • Tom Lane sent in another revision of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Fabien COELHO sent in another revision of a patch to improve pgbench syntax error messages.
  • Fabien COELHO sent in another revision of a patch to extend pgbench expressions with functions.
  • Pavel Stehule sent in another revision of a patch to add a row_to_array function.
  • Michael Paquier sent in a patch to describe precise rounding behavior of numeric and double precision in the docs.
  • Andres Freund sent in a patch to make heap extension scale better.

par N Bougain le samedi 11 avril 2015 à 16h27

jeudi 2 avril 2015

Dimitri Fontaine

pgDay Paris

Le 21 avril prochain se tient le premier pgDay Paris: une conférence PostgreSQL d'une journée complète. Il s'agit de 8 conférences sur votre base de données préférée par des conférencers internationaux, incluant des retours d'expérience et une analyse de l'utilisation des derniers développements en cours dans notre projet de base de données préféré.

pgDay Paris, 21 avril 2015 : 8 conférences PostgreSQL

Pour 65 € l'accès à la conférence inclut les pauses cafés (avec thé, eau, biscuits, etc) et le déjeuner sur place (repas complet servi à table).

Demandez le programme !

Le programme de pgDay.paris est publié et sa lecture permet de positionner pgDay Paris comme une conférence PostgreSQL incontournable en France.

Nous commencerons avec une présentation de l'utilisation de PostgreSQL dans le groupe ADEO, dont vous connaissez certainement plusieurs des enseignes : les magazins Leroy Merlin, Brico Center ou encore weldom font partie du groupe. Leurs logiciels de caisse utilisent PostgreSQL afin de ne jamais perdre une transaction de paiement !

Ensuite Stéphane Schildknecht nous détaillera un cas d'usage intéressant de PostgreSQL avec de la réplication Slony. Puis Magnus Hagander, committer PostgreSQL et membre de la Core Team du projet, viendra nous présenter les nouveautés de PostgreSQL 9.5, qui va bientôt entrer en période de béta.

Le projet Open Source ToroDB nous sera ensuite présenté par son auteur principal, Álvaro Hernandez. Il s'agit d'une implémentation complète du protocole MongoDB qui utilise PostgreSQL afin de gérer ses données. Pour tout ceux qui se posent la question du NoSQL à comparer au relationnel, ToroDB apporte une réponse intéressante tant du point de vue de l'utilisation en tant que développeur que des performances (volumes stockés, temps d'insertion ou manipulation des données, temps d'exécution des requêtes).

Ensuite Grégoire Hubert viendra lui-même présenter POMM, une approche moderne d'intégration de la base de données et du code orienté objet. Cette approche permet de dépasser les limitations classiques des ORMs et d'utiliser l'ensemble des fonctionnalités du moteur de base de données utilisé !

Nous parlerons égalemment des nouveaux projets de réplication intégrés à PostgreSQL, BDR pour Bi-Directional Replication. Développeur au coeur du projet, Petr Jelinek interviendra afin de montrer comment les solutions BDR et UDR peuvent être utilisées pour résoudre des problématiques classiques de Haute Disponibilité des services et des données avec PostgreSQL.

Les Systèmes d'information géographique seront à l'honneur avec Vincent Picavet, contributeur à PostGIS, qui viendra nous expliquer ce que l'on peut faire de PostgreSQL avec les extension GIS.

Nous terminerons cette journée riche en conférences avec une présentation des progrès de PostgreSQL dans le passé et à l'avenir, ainsi qu'une analyse du positionnement de notre Système de Gestion de Données Relationnelles préféré. Simon Riggs contribue à PostgreSQL depuis plus de 10 ans et a développé des fonctionnalités essentielles : Point In Time Recovery, Hot Standy, Synchronous Replication sont quelques exemples. Il dirige l'effort de développement de l'avenir de la réplication dans PostgreSQL avec Logical Decoding en 9.4 et Bi-Directional Replication. Committer PostgreSQL très actif, DBA certifié Oracle® et Teradata, Simon est le conférencier idéal lorsqu'il s'agit de positionner PostgreSQL dans le contexte actuel.

Conférence

La conférence pgDay Paris aura lieu le 21 avril prochain au siège de La Poste (les détails et la carte sont sur http://pgday.paris/contact/), et nous vous attendons nombreux : Inscrivez-vous dès aujourd'hui !

par dim@tapoueh.org (Dimitri Fontaine) le jeudi 2 avril 2015 à 09h50

mercredi 1 avril 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 22 mars 2015

La conférence PGDay UK aura lieu le 7 juillet 2015 – elle vise les membres de la communauté PostgreSQL anglaise. L'appel à conférenciers expire le 13 avril : http://www.postgresqlusergroup.org.uk

L'appel à conférenciers pour le PostgresOpen 2015, programmé à Dallas (Texas) du 16 au 18 septembre, a été lancé : http://2015.postgresopen.org/callforpapers/

[ndt: 4e rendez-vous du PLUG (Lyon) le 15 avril, avec une présentation de PoWA et des techniques de détection d'index manquants : http://www.meetup.com/PostgreSQL-User-Group-Lyon/events/221188759/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mars

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Replace insertion sort in contrib/intarray with qsort(). It's all very well to claim that a simplistic sort is fast in easy cases, but O(N^2) in the worst case is not good ... especially if the worst case is as easy to hit as "descending order input". Replace that bit with our standard qsort. Per bug #12866 from Maksym Boguk. Back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/8d1f239003d0245dda636dfa6cf0add13bee69d6
  • Allow foreign tables to participate in inheritance. Foreign tables can now be inheritance children, or parents. Much of the system was already ready for this, but we had to fix a few things of course, mostly in the area of planner and executor handling of row locks. As side effects of this, allow foreign tables to have NOT VALID CHECK constraints (and hence to accept ALTER ... VALIDATE CONSTRAINT), and to accept ALTER SET STORAGE and ALTER SET WITH/WITHOUT OIDS. Continuing to disallow these things would've required bizarre and inconsistent special cases in inheritance behavior. Since foreign tables don't enforce CHECK constraints anyway, a NOT VALID one is a complete no-op, but that doesn't mean we shouldn't allow it. And it's possible that some FDWs might have use for SET STORAGE or SET WITH OIDS, though doubtless they will be no-ops for most. An additional change in support of this is that when a ModifyTable node has multiple target tables, they will all now be explicitly identified in EXPLAIN output, for example: Update on pt1 (cost=0.00..321.05 rows=3541 width=46) Update on pt1 Foreign Update on ft1 Foreign Update on ft2 Update on child3 -> Seq Scan on pt1 (cost=0.00..0.00 rows=1 width=46) -> Foreign Scan on ft1 (cost=100.00..148.03 rows=1170 width=46) -> Foreign Scan on ft2 (cost=100.00..148.03 rows=1170 width=46) -> Seq Scan on child3 (cost=0.00..25.00 rows=1200 width=46) This was done mainly to provide an unambiguous place to attach "Remote SQL" fields, but it is useful for inherited updates even when no foreign tables are involved. Shigeru Hanada and Etsuro Fujita, reviewed by Ashutosh Bapat and Kyotaro Horiguchi, some additional hacking by me http://git.postgresql.org/pg/commitdiff/cb1ca4d800621dcae67ca6c799006de99fa4f0a5

Álvaro Herrera a poussé :

  • Fix out-of-array-bounds compiler warning. Since the array length check is using a post-increment operator, the compiler complains that there's a potential write to one element beyond the end of the array. This is not possible currently: the only path to this function is through pg_get_object_address(), which already verifies that the input array is no more than two elements in length. Still, a bug is a bug. No idea why my compiler doesn't complain about this ... Pointed out by Dead Rasheed and Peter Eisentraut http://git.postgresql.org/pg/commitdiff/a190738457353ddb60743e45972f6fe50a75ee77
  • Support opfamily members in get_object_address. In the spirit of 890192e99af and 4464303405f: have get_object_address understand individual pg_amop and pg_amproc objects. There is no way to refer to such objects directly in the grammar -- rather, they are almost always considered an integral part of the opfamily that contains them. (The only case that deals with them individually is ALTER OPERATOR FAMILY ADD/DROP, which carries the opfamily address separately and thus does not need it to be part of each added/dropped element's address.) In event triggers it becomes possible to become involved with individual amop/amproc elements, and this commit enables pg_get_object_address to do so as well. To make the overall coding simpler, this commit also slightly changes the get_object_address representation for opclasses and opfamilies: instead of having the AM name in the objargs array, I moved it as the first element of the objnames array. This enables the new code to use objargs for the type names used by pg_amop and pg_amproc. Reviewed by: Stephen Frost http://git.postgresql.org/pg/commitdiff/a61fd5334eb1040d0dcec0368702398a5b49152c
  • Rationalize vacuuming options and parameters. We were involving the parser too much in setting up initial vacuuming parameters. This patch moves that responsibility elsewhere to simplify code, and also to make future additions easier. To do this, create a new struct VacuumParams which is filled just prior to vacuum execution, instead of at parse time; for user-invoked vacuuming this is set up in a new function ExecVacuum, while autovacuum sets it up by itself. While at it, add a new member VACOPT_SKIPTOAST to enum VacuumOption, only set by autovacuum, which is used to disable vacuuming of the toast table instead of the old do_toast parameter; this relieves the argument list of vacuum() and some callees a bit. This partially makes up for having added more arguments in an effort to avoid having autovacuum from constructing a VacuumStmt parse node. Author: Michael Paquier. Some tweaks by Álvaro Reviewed by: Robert Haas, Stephen Frost, Álvaro Herrera http://git.postgresql.org/pg/commitdiff/0d831389749a3baaced7b984205b9894a82444b9
  • Setup cursor position for schema-qualified elements. This makes any errors thrown while looking up such schemas report the position of the error. Author: Ryan Kelly Reviewed by: Jeevan Chalke, Tom Lane http://git.postgresql.org/pg/commitdiff/b8d226b4f9691c7afb986dbaaf3f6ff7b203d1b5
  • Install shared libraries to bin/ in Windows under MSVC. Since commit cb4a3b04 we were already doing this for the Cygwin/mingw toolchains, but MSVC had not been updated to do it. At Install.pm time, the Makefile (or GNUmakefile) is inspected, and if a line matching SO_MAJOR_VERSION is found (indicating a shared library is being built), then files with the .dll extension are set to be installed in bin/ rather than lib/, while files with .lib extension are installed in lib/. This makes the MSVC toolchain up to date with cygwin/mingw. This removes ad-hoc hacks that were copying files into bin/ or lib/ manually (libpq.dll in particular was already being copied into bin). So while this is a rather ugly kludge, it's still cleaner than what was there before. Author: Michael Paquier Reviewed by: Asif Naeem http://git.postgresql.org/pg/commitdiff/f9dead5624c63b009fc04229c1e5f660436b747b
  • array_offset() and array_offsets(). These functions return the offset position or positions of a value in an array. Author: Pavel Stěhule Reviewed by: Jim Nasby http://git.postgresql.org/pg/commitdiff/13dbc7a824b3f905904cab51840d37f31a07a9ef

Andres Freund a poussé :

  • Remove docs missed in 51c11a7025. Somehow I misresolved a merge conflict when forward porting Petr's patch leading to a section of the docs remaining... Thankfully Fujii spotted my mistake. http://git.postgresql.org/pg/commitdiff/4559167c6b75be334fabad70d7cc03a38a08d494
  • Use 128-bit math to accelerate some aggregation functions. On platforms where we support 128bit integers, use them to implement faster transition functions for sum(int8), avg(int8), var_*(int2/int4),stdev_*(int2/int4). Where not supported continue to use numeric as a transition type. In some synthetic benchmarks this has been shown to provide significant speedups. Bumps catversion. Discussion: 544BB5F1.50709@proxel.se Author: Andreas Karlsson Reviewed-By: Peter Geoghegan, Petr Jelinek, Andres Freund, Oskari Saarenmaa, David Rowley http://git.postgresql.org/pg/commitdiff/959277a4f579da5243968c750069570a58e92b38
  • Add, optional, support for 128bit integers. We will, for the foreseeable future, not expose 128 bit datatypes to SQL. But being able to use 128bit math will allow us, in a later patch, to use 128bit accumulators for some aggregates; leading to noticeable speedups over using numeric. So far we only detect a gcc/clang extension that supports 128bit math, but no 128bit literals, and no *printf support. We might want to expand this in the future to further compilers; if there are any that that provide similar support. Discussion: 544BB5F1.50709@proxel.se Author: Andreas Karlsson, with significant editorializing by me Reviewed-By: Peter Geoghegan, Oskari Saarenmaa http://git.postgresql.org/pg/commitdiff/8122e1437e332e156d971a0274879b0ee76e488a
  • Fix minor copy & pasto in the int128 accumulator patch. It's unlikely that using PG_GETARG_INT16 instead of PG_GETARG_INT32 in this pace can cause actual problems, but this still should be fixed. http://git.postgresql.org/pg/commitdiff/59b0a98af07cf8decfe94739f92bf18ebb34ffc6

Bruce Momjian a poussé :

Robert Haas a poussé :

  • Fix status reporting for terminated bgworkers that were never started. Previously, GetBackgroundWorkerPid() would return BGWH_NOT_YET_STARTED if the slot used for the worker registration had not been reused by unrelated activity, and BGWH_STOPPED if it had. Either way, a process that had requested notification when the state of one of its background workers changed did not receive such notifications. Fix things so that GetBackgroundWorkerPid() always returns BGWH_STOPPED in this situation, so that we do not erroneously give waiters the impression that the worker will eventually be started; and send notifications just as we would if the process terminated after having been started, so that it's possible to wait for the postmaster to process a worker termination request without polling. Discovered by Amit Kapila during testing of parallel sequential scan. Analysis and fix by me. Back-patch to 9.4; there may not be anyone relying on this interface yet, but if anyone is, the new behavior is a clear improvement. http://git.postgresql.org/pg/commitdiff/bf740ce9e5d82612889d131f34c079215973ca00
  • Add flags argument to dsm_create. Right now, there's only one flag, DSM_CREATE_NULL_IF_MAXSEGMENTS, which suppresses the error that would normally be thrown when the maximum number of segments already exists, instead returning NULL. It might be useful to add more flags in the future, such as one to ignore allocation errors, but I haven't done that here. http://git.postgresql.org/pg/commitdiff/12968cf4085409c50f70c6643d92befdb34008f6

Stephen Frost a poussé :

  • GetUserId() changes to has_privs_of_role(). The pg_stat and pg_signal-related functions have been using GetUserId() instead of has_privs_of_role() for checking if the current user should be able to see details in pg_stat_activity or signal other processes, requiring a user to do 'SET ROLE' for inheirited roles for a permissions check, unlike other permissions checks. This patch changes that behavior to, instead, act like most other permission checks and use has_privs_of_role(), removing the 'SET ROLE' need. Documentation and error messages updated accordingly. Per discussion with Alvaro, Peter, Adam (though not using Adam's patch), and Robert. Reviewed by Jeevan Chalke. http://git.postgresql.org/pg/commitdiff/bf038899965263dbc4aef2b43c8fdfe6f49b788f

Peter Eisentraut a poussé :

Heikki Linnakangas a poussé :

  • Make pg_xlogdump MSVC build work more like others. Instead of copying xlogreader.c and *desc.c files into the source directory, build them where they are. That's what we do for other binaries that need to compile and link in files from elsewhere in the source tree. The commit history suggests that it was done this way because of issues with older versions of MSVC. I think this should work, but we'll see if the buildfarm complains. http://git.postgresql.org/pg/commitdiff/1933a5bbc883fd2697c42d82ae12f2d585559b23

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Kyotaro HORIGUCHI sent in another revision of a patch to reduce pinning in btree indexes.
  • Kaigai Kouhei sent in two more revisions of a patch to add custom/foreign join APIs.
  • David Rowley sent in another revision of a patch to allow removing inner JOINs under certain conditions.
  • Dean Rasheed sent in another revision of a patch to improve RLS qual pushdown.
  • Álvaro Herrera sent in another revision of a flock of patches to add deparsing utility commands.
  • Adam Brightwell sent in a patch to fix a regression test for sepgsql.
  • Michael Paquier sent in another revision of a patch to add in-core regression tests for recovery.
  • Álvaro Herrera sent in another revision of a patch to add a missing type_schema hint.
  • Álvaro Herrera sent in another revision of a patch to move freeze parameters of VacuumStmt into a separate code path.
  • Robert Haas sent in three more revisions of a patch to add parallel mode and parallel contexts.
  • Julien Tachoires and Andreas Karlsson traded patches to allow toast tables to be moved to a different tablespace.
  • Bruce Momjian and Tom Lane traded patches to prevent post-commit interrupts.
  • Robert Haas sent in another revision of a patch to allow testing parallel safety.
  • Amit Kapila sent in another revision of a patch to implement parallel sequential scan.
  • Michael Paquier sent in another revision of a patch to implement table-level log_autovacuum_min_duration.
  • Michael Paquier sent in a patch to fix an example of a variable referencing itself in example of pgbench.sgml.
  • David Christensen sent in two revisions of a patch to add a two-arg current_setting() with fallback.
  • Max Filippov sent in a patch to compare the linker and compiler outputs in the case of pthread with their default output.
  • Bruce Momjian sent in a patch to document the fact that repeatable read and serializable transactions see data committed after transaction start.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add shared infrastructure and functional dependencies for the upcoming multivariate statistics feature.
  • Bruce Momjian sent in another revision of a patch to change pg_ctl's default shutdown mode from "smart" to "fast."
  • Bruce Momjian sent in four revisions of a patch to simplify examples of dynamic SQL.
  • Bruce Momjian sent in a patch to remove an extra VACUUM in initdb.
  • Bruce Momjian sent in two more revisions of a patch to enable asciidoc as a psql output format.
  • David Wheeler sent in another revision of a patch to add launchd support.
  • Peter Geoghegan sent in two more revisions of a patch to add sort support for numerics.
  • Bruce Momjian sent in a patch to document more clearly the limitations of NUMERIC.
  • David Rowley sent in another revision of a patch to improve performance for joins where outer side is unique.
  • Peter Eisentraut sent in another revision of a patch to add TRANSFORMs.
  • Tomas Vondra sent in a patch to add two new log_line_prefix escape sequences - %T and %M, doing the same thing as %t and %m, but formatting the value as a number.
  • Andrew (RhodiumToad) Gierth sent in two more revisions of a patch to standardize INT64_MIN INT64_MAX UINT64_MAX.
  • Fabien COELHO sent in a patch to fix pgbench --progress report under (very) low rate.
  • Amit Khandekar sent in a patch to reset background worker crash state.
  • Dmitry Voronin sent in a patch to add some new functions to th SSL contrib module.
  • Pavel Stehule sent in another revision of a patch to add an ASSERT statement to SQL.

par N Bougain le mercredi 1 avril 2015 à 00h17

dimanche 29 mars 2015

Guillaume Lelarge

Version 0.2 de mon livre sur PostgreSQL

La version 0.2 de mon livre est sortie hier. En dehors des ajouts/correctifs dans les précédents chapitres, elle ajoute deux nouveaux chapitres.

Le premier concerne le protocole de communication client/serveur de PostgreSQL. Il permet de bien prendre conscience du dialogue et des possibilités d'échange entre ces deux entités. Le second aborde la question des connexions : comment s'établie une connexion, quels paramètres de configuration existent pour les connexions, comment gérer les connexions, etc.

De plus, le site d-booker, éditeur du livre, publie une interview de l'auteur (donc moi).

Si vous avez lu le livre « PostgreSQL, architectures et notions avancées », j'aimerais beaucoup savoir ce que vous en avez pensé. N'hésitez pas à intervenir sur le forum pour me remonter vos impressions ou tout problème que vous aurez constaté.

par Guillaume Lelarge le dimanche 29 mars 2015 à 16h46

mercredi 25 mars 2015

Rodolphe Quiédeville

PgDay Paris

J'ai la plaisir cette année de participer au comité de sélection du PgDay Paris qui se déroulera le 21 Avril 2015.

Le pgDay Paris est une journée de conférences et d'échanges organisée par la communauté française de PostgreSQL. Un ensemble de présentations en anglais et en français sera proposé, couvrant des sujets techniques ainsi que des retours d'expérience d'utilisation de PostgreSQL en production.

Que vous soyez développeur, administrateur système ou de bases de données ou bien « décideur » (DSI, directeur technique, etc), nous aurons du contenu pour vous !

Les inscriptions à l'événement sont dès maintenant disponibles pour le prix de 65 € pour la journée, qui inclut les pauses cafés et le déjeuner complet sur place.

Les places étant limitées, je vous invite à vous inscrire au plus tôt afin d'être sûr de pouvoir venir.

  • https://www.postgresql.eu/events/register/pgdayparis2015/

Le prix des places est maintenu volontairement bas afin de permettre au plus grand nombre de participer. Cela est rendu possible grâce au soutien des sponsors, et il reste là aussi des places. Alors si vous souhaitez apporter votre contribution au développement de PostgreSQL n'hésitez pas à prendre contact, toutes les coordonnées sont sur le site de l'évènement.

Rendez-vous le 21 !

par Rodolphe Quiédeville le mercredi 25 mars 2015 à 20h18

dimanche 22 mars 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 15 mars 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en Mars

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Fujii Masao a poussé :

  • Add missing "goto err" statements in xlogreader.c. Spotted by Andres Freund. http://git.postgresql.org/pg/commitdiff/c74c04b8aa03f05983f940ee94c86a5cc1945393
  • Fix typo in comment. http://git.postgresql.org/pg/commitdiff/828599acecdb2929f9b66d6f590c2abbc751b58b
  • Suppress maybe-uninitialized compiler warnings. Previously some compilers were thinking that the variables that 57aa5b2 added maybe-uninitialized. Spotted by Andres Freund http://git.postgresql.org/pg/commitdiff/cd6c45cbeec5e21b470e9a8d19e02f60f1a52807
  • Add GUC to enable compression of full page images stored in WAL. When newly-added GUC parameter, wal_compression, is on, the PostgreSQL server compresses a full page image written to WAL when full_page_writes is on or during a base backup. A compressed page image will be decompressed during WAL replay. Turning this parameter on can reduce the WAL volume without increasing the risk of unrecoverable data corruption, but at the cost of some extra CPU spent on the compression during WAL logging and on the decompression during WAL replay. This commit changes the WAL format (so bumping WAL version number) so that the one-byte flag indicating whether a full page image is compressed or not is included in its header information. This means that the commit increases the WAL volume one-byte per a full page image even if WAL compression is not used at all. We can save that one-byte by borrowing one-bit from the existing field like hole_offset in the header and using it as the flag, for example. But which would reduce the code readability and the extensibility of the feature. Per discussion, it's not worth paying those prices to save only one-byte, so we decided to add the one-byte flag to the header. This commit doesn't introduce any new compression algorithm like lz4. Currently a full page image is compressed using the existing PGLZ algorithm. Per discussion, we decided to use it at least in the first version of the feature because there were no performance reports showing that its compression ratio is unacceptably lower than that of other algorithm. Of course, in the future, it's worth considering the support of other compression algorithm for the better compression. Rahila Syed and Michael Paquier, reviewed in various versions by myself, Andres Freund, Robert Haas, Abhijit Menon-Sen and many others. http://git.postgresql.org/pg/commitdiff/57aa5b2bb11a4dbfdfc0f92370e0742ae5aa367b

Michael Meskes a poussé :

Heikki Linnakangas a poussé :

Robert Haas a poussé :

  • Fix handling of sortKeys field in Tuplesortstate. Commit 5cefbf5a6c4466ac6b1cc2a4316b4eba9108c802 introduced an assumption that this field would always be non-NULL when doing a merge pass, but that's not true. Without this fix, you can crash the server by building a hash index that is sufficiently large relative to maintenance_work_mem, or by triggering a large datum sort. Commit 5ea86e6e65dd2da3e9a3464484985d48328e7fe3 changed the comments for that field to say that it would be set in all cases except for the hash index case, but that wasn't (and still isn't) true. The datum-sort failure was spotted by Tomas Vondra; initial analysis of that failure was by Peter Geoghegan. The remaining issues were spotted by me during review of the surrounding code, and the patch is all my fault. http://git.postgresql.org/pg/commitdiff/2720e96a9bd58b2af177c714c0c5402773e1cf30
  • Allow named parameters to be specified using => in addition to := SQL has standardized on => as the use of to specify named parameters, and we've wanted for many years to support the same syntax ourselves, but this has been complicated by the possible use of => as an operator name. In PostgreSQL 9.0, we began emitting a warning when an operator named => was defined, and in PostgreSQL 9.2, we stopped shipping a =>(text, text) operator as part of hstore. By the time the next major version of PostgreSQL is released, => will have been deprecated for a full five years, so hopefully there won't be too many people still relying on it. We continue to support := for compatibility with previous PostgreSQL releases. Pavel Stehule, reviewed by Petr Jelinek, with a few documentation tweaks by me. http://git.postgresql.org/pg/commitdiff/865f14a2d31af23a05bbf2df04c274629c5d5c4d
  • Suggest to the user the column they may have meant to reference. Error messages informing the user that no such column exists can sometimes provoke a perplexed response. This often happens due to a subtle typo in the column name or, perhaps less likely, in the alias name. To speed discovery of what the real issue is in such cases, we'll now search the range table for approximate matches. If there are one or two such matches that are good enough to think that they might be what the user intended to type, and better than all other approximate matches, we'll issue a hint suggesting that the user might have intended to reference those columns. Peter Geoghegan and Robert Haas http://git.postgresql.org/pg/commitdiff/e529cd4ffa605c6f14f1391af5559b3a44da0336
  • sepgsql: Improve error message when unsupported object type is labeled. KaiGai Kohei, reviewed by Álvaro Herrera and myself http://git.postgresql.org/pg/commitdiff/e96b7c6b9fc4d148a22588894245416b63743368
  • Require non-NULL pstate for all addRangeTableEntryFor* functions. Per discussion, it's better to have a consistent coding rule here. Michael Paquier, per a node from Greg Stark referencing an old post from Tom Lane. http://git.postgresql.org/pg/commitdiff/bc93ac12c2544b6b3a68b6cb0282e0828fa14a34
  • Document the new custom scan APIs. These APIs changed somewhat subsequent to the initial commit, and may change further in the future, but let's document what we have today. KaiGai Kohei and Robert Haas, reviewed by Tom Lane and Thom Brown http://git.postgresql.org/pg/commitdiff/82fe8b1119e4187f3d991564274607b0b4089aca

Álvaro Herrera a poussé :

  • Allow CURRENT/SESSION_USER to be used in certain commands. Commands such as ALTER USER, ALTER GROUP, ALTER ROLE, GRANT, and the various ALTER OBJECT / OWNER TO, as well as ad-hoc clauses related to roles such as the AUTHORIZATION clause of CREATE SCHEMA, the FOR clause of CREATE USER MAPPING, and the FOR ROLE clause of ALTER DEFAULT PRIVILEGES can now take the keywords CURRENT_USER and SESSION_USER as user specifiers in place of an explicit user name. This commit also fixes some quite ugly handling of special standards- mandated syntax in CREATE USER MAPPING, which in particular would fail to work in presence of a role named "current_user". The special role specifiers PUBLIC and NONE also have more consistent handling now. Also take the opportunity to add location tracking to user specifiers. Authors: Kyotaro Horiguchi. Heavily reworked by Álvaro Herrera. Reviewed by: Rushabh Lathia, Adam Brightwell, Marti Raudsepp. http://git.postgresql.org/pg/commitdiff/31eae6028eca4365e7165f5f33fee1ed0486aee0
  • Fix crasher bugs in previous commit. ALTER DEFAULT PRIVILEGES was trying to decode the list of roles in the FOR clause as a list of names rather than of RoleSpecs; and the IN clause in CREATE ROLE was doing the same thing. This was evidenced by crashes on some buildfarm machines, though on my platform this doesn't cause a failure by mere chance; I can reproduce the failures only by adding some padding in struct RoleSpecs. Fix by dereferencing those lists as being of RoleSpecs, not string Values. http://git.postgresql.org/pg/commitdiff/e3f1c24b992acb88e4ccf33118640aee4b11dd47
  • Keep CommitTs module in sync in standby and master. We allow this module to be turned off on restarts, so a restart time check is enough to activate or deactivate the module; however, if there is a standby replaying WAL emitted from a master which is restarted, but the standby isn't, the state in the standby becomes inconsistent and can easily be crashed. Fix by activating and deactivating the module during WAL replay on parameter change as well as on system start. Problem reported by Fujii Masao in http://www.postgresql.org/message-id/CAHGQGwFhJ3CnHo1CELEfay18yg_RA-XZT-7D8NuWUoYSZ90r4Q@mail.gmail.com Author: Petr Jelínek http://git.postgresql.org/pg/commitdiff/4f3924d9cd438ba4e6fd639460f8c859c65d45a3
  • Fix stray sentence fragment in shared_preload_libraries documentation. The introduction in the Shared Library Preloading section already instructs the user to separate multiple library names with commas, so just remove the fragment from here. Author: Dagfinn Ilmari Mannsåker http://git.postgresql.org/pg/commitdiff/bb7b35caf78de80d2ff1643d042e62a71f83abbb
  • Move BRIN page type to page's last two bytes. ... which is the usual convention among AMs, so that pg_filedump and similar utilities can tell apart pages of different AMs. It was also the intent of the original code, but I failed to realize that alignment considerations would move the whole thing to the previous-to-last word in the page. The new definition of the associated macro makes surrounding code a bit leaner, too. Per note from Heikki at http://www.postgresql.org/message-id/546A16EF.9070005@vmware.com http://git.postgresql.org/pg/commitdiff/e491bd2ee34860b14ff18abc5602f9aa5b197a2d
  • Refactor Mkvcbuild.pm to facilitate modules migrations. This is in preparation to "upgrade" some modules from contrib/ to src/bin/, per discussion. Author: Michael Paquier http://git.postgresql.org/pg/commitdiff/66ece312f99f384bd33e4342580e78b0eebf0e74
  • Support user mappings in get_object_address. Since commit 72dd233d3ef we were trying to obtain object addressing information in sql_drop event triggers, but that caused failures when the drops involved user mappings. This addition enables that to work again. Naturally, pg_get_object_address can work with these objects now, too. I toyed with the idea of removing DropUserMappingStmt as a node and using DropStmt instead in the DropUserMappingStmt grammar production, but that didn't go very well: for one thing the messages thrown by the specific code are specialized (you get "server not found" if you specify the wrong server, instead of a generic "user mapping for ... not found" which you'd get it we were to merge this with RemoveObjects --- unless we added even more special cases). For another thing, it would require to pass RoleSpec nodes through the objname/objargs representation used by RemoveObjects, which works in isolation, but gets messy when pg_get_object_address is involved. So I dropped this part for now. Reviewed by Stephen Frost. http://git.postgresql.org/pg/commitdiff/890192e99af5db1d15d5bb73f3f1044faa1d2758
  • Fix libpq test expected output file. Evidently, this test is not run very frequently ... http://git.postgresql.org/pg/commitdiff/d4d7777548ed3ea2ca579003e37f9df4d0e0ab9e
  • Support default ACLs in get_object_address. In the spirit of 890192e99af, this time add support for the things living in the pg_default_acl catalog. These are not really "objects", but they show up as such in event triggers. There is no "DROP DEFAULT PRIVILEGES" or similar command, so it doesn't look like the new representation given would be useful anywhere else, so I didn't try to use it outside objectaddress.c. (That might be a bug in itself, but that would be material for another commit.) Reviewed by Stephen Frost. http://git.postgresql.org/pg/commitdiff/4464303405f1f886d63f8316386621cd7436c5d6

Tom Lane a poussé :

  • Clean up the mess from => patch. Commit 865f14a2d31af23a05bbf2df04c274629c5d5c4d was quite a few bricks shy of a load: psql, ecpg, and plpgsql were all left out-of-step with the core lexer. Of these only the last was likely to be a fatal problem; but still, a minimal amount of grepping, or even just reading the comments adjacent to the places that were changed, would have found the other places that needed to be changed. http://git.postgresql.org/pg/commitdiff/2fbb286647fac2014abdf2fbf6c7b4134be91602
  • Allocate ParamListInfo once per plpgsql function, not once per expression. setup_param_list() was allocating a fresh ParamListInfo for each query or expression evaluation requested by a plpgsql function. There was probably once good reason to do it like that, but for a long time we've had a convention that there's a one-to-one mapping between the function's PLpgSQL_datum array and the ParamListInfo slots, which means that a single ParamListInfo can serve all the function's evaluation requests: the data that would need to be passed is the same anyway. In this patch, we retain the pattern of zeroing out the ParamListInfo contents during each setup_param_list() call, because some of the slots may be stale and we don't know exactly which ones. So this patch only saves a palloc/pfree per evaluation cycle and nothing more; still, that seems to be good for a couple percent overall speedup on simple-arithmetic type statements. In future, though, we might be able to improve matters still more by managing the param array contents more carefully. Also, unify the former use of estate->cur_expr with that of paramLI->parserSetupArg; they both were used to point to the active expression, so we can combine the variables into just one. http://git.postgresql.org/pg/commitdiff/21dcda2713656a7483e3280ac9d2ada20a87a9a9
  • Make operator precedence follow the SQL standard more closely. While the SQL standard is pretty vague on the overall topic of operator precedence (because it never presents a unified BNF for all expressions), it does seem reasonable to conclude from the spec for <boolean value expression> that OR has the lowest precedence, then AND, then NOT, then IS tests, then the six standard comparison operators, then everything else (since any non-boolean operator in a WHERE clause would need to be an argument of one of these). We were only sort of on board with that: most notably, while "<" ">" and "=" had properly low precedence, "<=" ">=" and "<>" were treated as generic operators and so had significantly higher precedence. And "IS" tests were even higher precedence than those, which is very clearly wrong per spec. Another problem was that "foo NOT SOMETHING bar" constructs, such as "x NOT LIKE y", were treated inconsistently because of a bison implementation artifact: they had the documented precedence with respect to operators to their right, but behaved like NOT (i.e., very low priority) with respect to operators to their left. Fixing the precedence issues is just a small matter of rearranging the precedence declarations in gram.y, except for the NOT problem, which requires adding an additional lookahead case in base_yylex() so that we can attach a different token precedence to NOT LIKE and allied two-word operators. The bulk of this patch is not the bug fix per se, but adding logic to parse_expr.c to allow giving warnings if an expression has changed meaning because of these precedence changes. These warnings are off by default and are enabled by the new GUC operator_precedence_warning. It's believed that very few applications will be affected by these changes, but it was agreed that a warning mechanism is essential to help debug any that are. http://git.postgresql.org/pg/commitdiff/c6b3c939b7e0f1d35f4ed4996e71420a993810d2
  • Improve planner's cost estimation in the presence of semijoins. If we have a semijoin, say SELECT * FROM x WHERE x1 IN (SELECT y1 FROM y) and we're estimating the cost of a parameterized indexscan on x, the number of repetitions of the indexscan should not be taken as the size of y; it'll really only be the number of distinct values of y1, because the only valid plan with y on the outside of a nestloop would require y to be unique-ified before joining it to x. Most of the time this doesn't make that much difference, but sometimes it can lead to drastically underestimating the cost of the indexscan and hence choosing a bad plan, as pointed out by David Kubečka. Fixing this is a bit difficult because parameterized indexscans are costed out quite early in the planning process, before we have the information that would be needed to call estimate_num_groups() and thereby estimate the number of distinct values of the join column(s). However we can move the code that extracts a semijoin RHS's unique-ification columns, so that it's done in initsplan.c rather than on-the-fly in create_unique_path(). That shouldn't make any difference speed-wise and it's really a bit cleaner too. The other bit of information we need is the size of the semijoin RHS, which is easy if it's a single relation (we make those estimates before considering indexscan costs) but problematic if it's a join relation. The solution adopted here is just to use the product of the sizes of the join component rels. That will generally be an overestimate, but since estimate_num_groups() only uses this input as a clamp, an overestimate shouldn't hurt us too badly. In any case we don't allow this new logic to produce a value larger than we would have chosen before, so that at worst an overestimate leaves us no wiser than we were before. http://git.postgresql.org/pg/commitdiff/b55722692ba0ceb934bb32bcddb562e2120f43dd
  • Fix old bug in get_loop_count(). While poking at David Kubečka's issue I noticed an ancient logic error in get_loop_count(): it used 1.0 as a "no data yet" indicator, but since that is actually a valid rowcount estimate, this doesn't work. If we have one input relation with 1.0 as rowcount and then another one with a larger rowcount, we should use 1.0 as the result, but we picked the larger rowcount instead. (I think when I coded this, I recognized the conflict, but mistakenly thought that the logic would pick the desired count anyway.) Fixing this changed the plan for one existing regression test case. Since the point of that test is to exercise creation of a particular shape of nestloop plan, I tweaked the query a little bit so it still results in the same plan choice. This is definitely a bug, but I'm hesitant to back-patch since it might change plan choices unexpectedly, and anyway failure to implement a heuristic precisely as intended is a pretty low-grade bug. http://git.postgresql.org/pg/commitdiff/b746d0c32d4fe749c8d39ccb09d8f0fb38bcc197
  • Support flattening of empty-FROM subqueries and one-row VALUES tables. We can't handle this in the general case due to limitations of the planner's data representations; but we can allow it in many useful cases, by being careful to flatten only when we are pulling a single-row subquery up into a FROM (or, equivalently, inner JOIN) node that will still have at least one remaining relation child. Per discussion of an example from Kyotaro Horiguchi. http://git.postgresql.org/pg/commitdiff/f4abd0241de20d5d6a79b84992b9e88603d44134
  • Ensure tableoid reads correctly in EvalPlanQual-manufactured tuples. The ROW_MARK_COPY path in EvalPlanQualFetchRowMarks() was just setting tableoid to InvalidOid, I think on the assumption that the referenced RTE must be a subquery or other case without a meaningful OID. However, foreign tables also use this code path, and they do have meaningful table OIDs; so failure to set the tuple field can lead to user-visible misbehavior. Fix that by fetching the appropriate OID from the range table. There's still an issue about whether CTID can ever have a meaningful value in this case; at least with postgres_fdw foreign tables, it does. But that is a different problem that seems to require a significantly different patch --- it's debatable whether postgres_fdw really wants to use this code path at all. Simplified version of a patch by Etsuro Fujita, who also noted the problem to begin with. The issue can be demonstrated in all versions having FDWs, so back-patch to 9.1. http://git.postgresql.org/pg/commitdiff/443fd0540e298b621be6748dead1fb444556e0d0
  • Improve documentation of bt_page_items(). Explain some of the funny conventions used in btree page items. Peter Geoghegan and Jeff Janes http://git.postgresql.org/pg/commitdiff/ebc0f5e01d2f4b7d7c3307fd4d40498f6b120898
  • Remove workaround for ancient incompatibility between readline and libedit. GNU readline defines the return value of write_history() as "zero if OK, else an errno code". libedit's version of that function used to have a different definition (to wit, "-1 if error, else the number of lines written to the file"). We tried to work around that by checking whether errno had become nonzero, but this method has never been kosher according to the published API of either library. It's reportedly completely broken in recent Ubuntu releases: psql bleats about "No such file or directory" when saving ~/.psql_history, even though the write worked fine. However, libedit has been following the readline definition since somewhere around 2006, so it seems all right to finally break compatibility with ancient libedit releases and trust that the return value is what readline specifies. (I'm not sure when the various Linux distributions incorporated this fix, but I did find that OS X has been shipping fixed versions since 10.5/Leopard.) If anyone is still using such an ancient libedit, they will find that psql complains it can't write ~/.psql_history at exit, even when the file was written correctly. This is no worse than the behavior we're fixing for current releases. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/df9ebf1eeaf98481e00cd77bf6056d42310731b7
  • Build src/port/dirmod.c only on Windows. Since commit ba7c5975adea74c6f17bdb0e0427ad85962092a2, port/dirmod.c has contained only Windows-specific functions. Most platforms don't seem to mind uselessly building an empty file, but OS X for one issues warnings. Hence, treat dirmod.c as a Windows-specific file selected by configure rather than one that's always built. We can revert this change if dirmod.c ever gains any non-Windows functionality again. Back-patch to 9.4 where the mentioned commit appeared. http://git.postgresql.org/pg/commitdiff/91f4a5a976500517e492320e389342d7436cf9d4
  • Remove obsolete comment. Obsoleted by commit 21dcda2713656a7483e3280ac9d2ada20a87a9a9, but I missed seeing the cross-reference in the comments for exec_eval_integer(). Also improve the cross-reference in the comments for exec_eval_cleanup(). http://git.postgresql.org/pg/commitdiff/5ff683962ef9a953eeb17ea58d678f0c4ca189ae
  • Add missing documentation for PGC_SU_BACKEND in description of pg_settings. Commit fe550b2ac249af5fbd8e9e19290a4ba43c882f2d missed updating this list of the PGC_XXX values, which in hindsight is not so surprising because catalogs.sgml is not a place you'd think to look for them. In addition to adding the missing doco, insert the PGC_XXX C enum names in SGML comments, so that grepping for the enum names will find this file. That might spare the next person similar embarrassment. Spotted by Magnus Hagander. http://git.postgresql.org/pg/commitdiff/d1e9214e4ff5604357d0155467f105825a9e102c
  • src/port/dirmod.c needs to be built on Cygwin too. Oversight in my commit 91f4a5a976500517e492320e389342d7436cf9d4. Per buildfarm member brolga. http://git.postgresql.org/pg/commitdiff/80089597730f67927293c410914f3e6bf11ca447
  • Move LockClauseStrength, LockWaitPolicy into new file nodes/lockoptions.h. Commit df630b0dd5ea2de52972d456f5978a012436115e moved enum LockWaitPolicy into its very own header file utils/lockwaitpolicy.h, which does not seem like a great idea from here. First, it's still a node-related declaration, and second, a file named like that can never sensibly be used for anything else. I do not think we want to encourage a one-typedef-per-header-file approach. The upcoming foreign table inheritance patch was doubling down on this bad idea by moving enum LockClauseStrength into its *own* can-never-be-used-for-anything-else file. Instead, let's put them both in a file named nodes/lockoptions.h. (They do seem to need a separate header file because we need them in both parsenodes.h and plannodes.h, and we don't want either of those including the other. Past practice might suggest adding them to nodes/nodes.h, but they don't seem sufficiently globally useful to justify that.) Committed separately since there's no functional change here, just some header-file refactoring. http://git.postgresql.org/pg/commitdiff/9fac5fd741ec17ae24dde6b8e82064f13c148ddf
  • Improve representation of PlanRowMark. This patch fixes two inadequacies of the PlanRowMark representation. First, that the original LockingClauseStrength isn't stored (and cannot be inferred for foreign tables, which always get ROW_MARK_COPY). Since some PlanRowMarks are created out of whole cloth and don't actually have an ancestral RowMarkClause, this requires adding a dummy LCS_NONE value to enum LockingClauseStrength, which is fairly annoying but the alternatives seem worse. This fix allows getting rid of the use of get_parse_rowmark() in FDWs (as per the discussion around commits 462bd95705a0c23b and 8ec8760fc87ecde0), and it simplifies some things elsewhere. Second, that the representation assumed that all child tables in an inheritance hierarchy would use the same RowMarkType. That's true today but will soon not be true. We add an "allMarkTypes" field that identifies the union of mark types used in all a parent table's children, and use that where appropriate (currently, only in preprocess_targetlist()). In passing fix a couple of minor infelicities left over from the SKIP LOCKED patch, notably that _outPlanRowMark still thought waitPolicy is a bool. Catversion bump is required because the numeric values of enum LockingClauseStrength can appear in on-disk rules. Extracted from a much larger patch to support foreign table inheritance; it seemed worth breaking this out, since it's a separable concern. Shigeru Hanada and Etsuro Fujita, somewhat modified by me http://git.postgresql.org/pg/commitdiff/7b8b8a43317e9e59eca8b511b714a0ab7da5f1cb

Andres Freund a poussé :

  • Add macros wrapping all usage of gcc's __attribute__. Until now __attribute__() was defined to be empty for all compilers but gcc. That's problematic because it prevents using it in other compilers; which is necessary e.g. for atomics portability. It's also just generally dubious to do so in a header as widely included as c.h. Instead add pg_attribute_format_arg, pg_attribute_printf, pg_attribute_noreturn macros which are implemented in the compilers that understand them. Also add pg_attribute_noreturn and pg_attribute_packed, but don't provide fallbacks, since they can affect functionality. This means that external code that, possibly unwittingly, relied on __attribute__ defined to be empty on !gcc compilers may now run into warnings or errors on those compilers. But there shouldn't be many occurances of that and it's hard to work around... Discussion: 54B58BA3.8040302@ohmu.fi Author: Oskari Saarenmaa, with some minor changes by me. http://git.postgresql.org/pg/commitdiff/bbfd7edae5aa5ad5553d3c7e102f2e450d4380d4
  • Adjust valgrind suppressions wrt 025c02420. http://git.postgresql.org/pg/commitdiff/241f088f3632814fe9dbd5bcbc509ec42a268d01
  • Increase max_wal_size's default from 128MB to 1GB. The introduction of min_wal_size & max_wal_size in 88e982302684 makes it feasible to increase the default upper bound in checkpoint size. Previously raising the default would lead to a increased disk footprint, even if more segments weren't beneficial. The low default of checkpoint size is one of common performance problem users have thus increasing the default makes sense. Setups where the increase in maximum disk usage is a problem will very likely have to run with a modified configuration anyway. Discussion: 54F4EFB8.40202@agliodbs.com, CA+TgmoZEAgX5oMGJOHVj8L7XOkAe05Gnf45rP40m-K3FhZRVKg@mail.gmail.com Author: Josh Berkus, after a discussion involving lots of people. http://git.postgresql.org/pg/commitdiff/a0f5954af19ddcfea946b15744f2006a789dc4bd
  • Remove pause_at_recovery_target recovery.conf setting. The new recovery_target_action (introduced in aedccb1f6/b8e33a85d4) replaces it's functionality. Having both seems likely to cause more confusion than it saves worry due to the incompatibility. Discussion: 5484FC53.2060903@2ndquadrant.com Author: Petr Jelinek http://git.postgresql.org/pg/commitdiff/51c11a7025ecc10c2b922d60a056ad7c6cf147a5
  • Merge the various forms of transaction commit & abort records. Since 465883b0a two versions of commit records have existed. A compact version that was used when no cache invalidations, smgr unlinks and similar were needed, and a full version that could deal with all that. Additionally the full version was embedded into twophase commit records. That resulted in a measurable reduction in the size of the logged WAL in some workloads. But more recently additions like logical decoding, which e.g. needs information about the database something was executed on, made it applicable in fewer situations. The static split generally made it hard to expand the commit record, because concerns over the size made it hard to add anything to the compact version. Additionally it's not particularly pretty to have twophase.c insert RM_XACT records. Rejigger things so that the commit and abort records only have one form each, including the twophase equivalents. The presence of the various optional (in the sense of not being in every record) pieces is indicated by a bits in the 'xinfo' flag. That flag previously was not included in compact commit records. To prevent an increase in size due to its presence, it's only included if necessary; signalled by a bit in the xl_info bits available for xact.c, similar to heapam.c's XLOG_HEAP_OPMASK/XLOG_HEAP_INIT_PAGE. Twophase commit/aborts are now the same as their normal counterparts. The original transaction's xid is included in an optional data field. This means that commit records generally are smaller, except in the case of a transaction with subtransactions, but no other special cases; the increase there is four bytes, which seems acceptable given that the more common case of not having subtransactions shrank. The savings are especially measurable for twophase commits, which previously always used the full version; but will in practice only infrequently have required that. The motivation for this work are not the space savings and and deduplication though; it's that it makes it easier to extend commit records with additional information. That's just a few lines of code now; without impacting the common case where that information is not needed. Discussion: 20150220152150.GD4149@awork2.anarazel.de, 235610.92468.qm%40web29004.mail.ird.yahoo.com Reviewed-By: Heikki Linnakangas, Simon Riggs http://git.postgresql.org/pg/commitdiff/4f1b890b1377744688e43218f975d3c8b2ae39f8

Peter Eisentraut a poussé :

Tatsuo Ishii a poussé :

  • Fix integer overflow in debug message of walreceiver. The message tries to tell the replication apply delay which fails if the first WAL record is not applied yet. Fix is, instead of telling overflowed minus numeric, showing "N/A" which indicates that the delay data is not yet available. Problem reported by me and patch by Fabrízio de Royes Mello. Back patched to 9.4, 9.3 and 9.2 stable branches (9.1 and 9.0 do not have the debug message). http://git.postgresql.org/pg/commitdiff/364c006c1fba7ba7825fb06ef0166e752546f357

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in two more revisions of a patch to speed up make check-world.
  • Dmitry Voronin sent in another revision of a patch to add some new functions to contrib/sslinfo.
  • Michael Paquier sent in two more revisions of a patch to install shared libs in lib/ and bin/ with MSVC (Was: install libpq.dll in bin directory on Windows / Cygwin.
  • Kaigai Kouhei sent in another revision of a patch to add join push-down support for foreign tables.
  • SAWADA Masahiko sent in three more revisions of a patch to add REINDEX xxx VERBOSE.
  • Andrew (RhodiumToad) Gierth sent in two more revisions of a patch to add GROUPING SETS.
  • Amit Kapila and Amit Langote traded patches to add parallel sequential scan.
  • Kevin Grittner sent in three more revisions of a patch to reduce pinning in btree indexes.
  • Kaigai Kouhei sent in another revision of a patch to add a custom plan API.
  • Kaigai Kouhei and Robert Haas traded patches for the custom join API.
  • Heikki Linnakangas sent in three more revisions of a patch to add pg_rewind.
  • Andreas Karlsson sent in three more revisions of a patch to use 128-bit integers for sum, avg and statistics aggregates.
  • Abhijit Menon-Sen sent in another revision of a patch to recursively fsync PGDATA at startup if needed.
  • Kyotaro HORIGUCHI sent in another revision of a patch to make an effort to send feedback regulary on heavy load.
  • Tom Lane sent in a patch to fix some inefficiencies in PL/pgsql.
  • Robert Haas sent in another revision of a patch to do better at HINTing at a missing column.
  • Michael Paquier and Peter Eisentraut traded flocks of patches intended to move binaries from contrib/ to bin/.
  • Jim Nasby and Pavel Stehule traded patches to add an array_offset function.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add regrole.
  • SAWADA Masahiko sent in another revision of a patch to add a pg_file_settings view.
  • Fabien COELHO sent in another revision of a patch to improve pgbench syntax error messages.
  • Gregory Stark sent in another revision of a patch to provide a catalog view to pg_hba.conf file.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add alter user/role CURRENT_USER.
  • Jozef Mlich sent in a patch to use the tcl binary in the PL/tcl scripts rather than invoking them as shell programs.
  • Marko (johto) Tiikkaja sent in a patch to fix a bug in dumping views with circular dependencies.
  • Stas Kelvich sent in another revision of a patch to add cube extension KNN support.
  • Álvaro Herrera sent in a patch to support opfamily members in get_object_address.
  • Robert Haas sent in another revision of a patch to help assessing parallel-safety.
  • Michael Paquier sent in another revision of a patch to fix the slash leanings in the MSVC build code.
  • Michael Paquier sent in a patch to move freeze parameters of VacuumStmt into a separate structure.
  • Tomas Vondra sent in a patch to fix an xloginsert.c hole_length warning on gcc 4.8.3.
  • Peter Geoghegan sent in another revision of a patch to do datumsort revisions for abbreviated keys.
  • Julien Tachoires sent in two more revisions of a patch to allow toast tables to be moved to a different tablespace.
  • David Rowley sent in another revision of a patch to improve performance for for joins where outer side is unique.
  • Pavel Stehule sent in a patch to add a --strict tablename parameter to pg_dump.
  • Andreas Karlsson sent in a patch to fix support for the --no-tablespaces flag for pg_dump.
  • Tom Lane sent in a patch to remove setup_param_list() overhead for the common case of PLPGSQL_DTYPE_VAR variables (ie, any non-composite type).
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • Andres Freund sent in another revision of a patch to merge compact/non compact commits, and make aborts dynamically sized.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to create a sequence access method.

par N Bougain le dimanche 22 mars 2015 à 22h28

Adrien Nayrat

Restauration PITR – Part 7

Je profite d’un week end de mauvais temps (m’empêchant de retourner en falaise :( ) pour poursuivre mes articles sur Postgres.

Dans le précédent article : Backup, restauration – Part 6 J’ai évoqué la restauration PITR pour Point In Time Recovery. Je vous encourage vivement à mettre en place ce qui va suivre afin de réduire la perte de données en cas de modification accidentelle sur des enregistrements : Un drop table par exemple.

Pour la mettre en place il vous faut :

  • Une instance Postgres maître.
  • Archivage des WAL, externalisé de préférence.

Je ne vais pas revenir sur la mise en place de l’archivage, je l’ai déjà détaillé dans les précédents articles. Pour vous expliquer cette restauration je vais utiliser un scénario “catastrophe”.

Il est 12H30 et on vous informe qu’une table a été supprimée par erreur sur un serveur de production.

La réplication que vous avez mis en place ne vous est d’aucune aide car l’opération a été reportée sur le réplicat.

Remarque : Avec la version 9.4 il est possible d’avoir un réplicat avec du retard. Avec un peu de chance l’opération de suppression de la table n’est peut être pas encore réalisée sur le réplicat.

Par chance, vous effectuez un backup full toutes les nuits à 1H00 et vous avez mis en place l’archivage des journaux. Leur nettoyage se fait après le backup full grâce à pg_archivecleanup.

Pour être tranquille vous effectuez un checkpoint afin que les derniers journaux soient archivés. Ensuite vous stoppez votre instance Postgres et déplacez le répertoire (au cas où).

Vous restaurez les fichiers du backup full. Il ne vous reste plus qu’a configurer correctement la restauration via le fichier recovery.conf.

Celui-ci devra contenir le paramètre restore_command afin que Postgres récupére les journaux de transactions archivés.

Il devra également contenir un paramètre pour la cible de récupération, Si on ne le spécifie pas le moteur va rejouer les journaux de transaction dans leur totalité, jusqu’au “drop table” fatal. Il est possible de demander la restauration jusqu’à un point de restauration, une date ou un identifiant de transaction.

Pour approfondir : http://docs.postgresqlfr.org/current/recovery-target-settings.html

Revenons à notre scénario, imaginons que nous souhaitons revenir à la dernière transaction jouée avant le drop fatal. En grattant un peu, on peut utiliser l’outil  pg_xlogdump pour lire les journaux de transaction.

/usr/lib/postgresql/9.3/bin/pg_xlogdump 000000010000000300000066
rmgr: Heap        len (rec/tot):     21/   197, tx:    1213547, lsn: 3/66000028, prev 3/650000F0, bkp: 1000, desc: insert: rel 1663/16486/16519; tid 0/2
rmgr: Transaction len (rec/tot):     12/    44, tx:    1213547, lsn: 3/660000F0, prev 3/66000028, bkp: 0000, desc: commit: 2015-03-22 12:23:35.834785 CET
rmgr: Standby     len (rec/tot):     16/    48, tx:    1213548, lsn: 3/66000120, prev 3/660000F0, bkp: 0000, desc: AccessExclusive locks: xid 1213548 db 16486 rel 16519
rmgr: Standby     len (rec/tot):     16/    48, tx:    1213548, lsn: 3/66000150, prev 3/66000120, bkp: 0000, desc: AccessExclusive locks: xid 1213548 db 16486 rel 16522
rmgr: Standby     len (rec/tot):     16/    48, tx:    1213548, lsn: 3/66000180, prev 3/66000150, bkp: 0000, desc: AccessExclusive locks: xid 1213548 db 16486 rel 16524
rmgr: Heap        len (rec/tot):     26/  6642, tx:    1213548, lsn: 3/660001B0, prev 3/66000180, bkp: 1000, desc: delete: rel 1663/16486/11774; tid 7/35 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  3270, tx:    1213548, lsn: 3/66001BA8, prev 3/660001B0, bkp: 1000, desc: delete: rel 1663/16486/11903; tid 46/44 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  4774, tx:    1213548, lsn: 3/66002888, prev 3/66001BA8, bkp: 1000, desc: delete: rel 1663/16486/11821; tid 2/26 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  5730, tx:    1213548, lsn: 3/66003B30, prev 3/66002888, bkp: 1000, desc: delete: rel 1663/16486/11786; tid 42/37 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/    58, tx:    1213548, lsn: 3/660051B0, prev 3/66003B30, bkp: 0000, desc: delete: rel 1663/16486/11786; tid 42/38 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  7902, tx:    1213548, lsn: 3/660051F0, prev 3/660051B0, bkp: 1000, desc: delete: rel 1663/16486/11797; tid 0/80 KEYS_UPDATED

C’est un extrait du résultat de la commande, le première ligne indique un insert avec la transaction N° 1213547, la ligne ne dessous indique un commit. En revanche à la troisième ligne on est passé à la transaction 1213548 avec un lock suivi de plusieurs lignes indiquant un delete. Il s’agit du drop fatal.

On va donc revenir à la transaction N°1213547. Voici le contenu de mon fichier recovery.conf :

restore_command = 'rsync "192.168.1.128:/var/lib/postgresql/9.3/wal_archive/%f" "%p"'

recovery_target_xid = '1213547'
recovery_target_inclusive = 'true'

La directive recovery_target_inclusive indique au moteur s’il inclue la transaction correspondant au N°1213547 ou s’il s’arrête juste avant. Ainsi j’aurais pu spécifier un xid de 1213548  et un target_inclusive à “false” pour obtenir le même résultat.

Ensuite il suffit de redémarrer postgres, voici ce qu’on peut observer dans les logs :

2015-03-22 12:41:17 CET LOG:  database system was interrupted; last known up at 2015-03-22 12:22:59 CET
2015-03-22 12:41:17 CET LOG:  creating missing WAL directory "pg_xlog/archive_status"
2015-03-22 12:41:17 CET LOG:  starting point-in-time recovery to XID 1213547
2015-03-22 12:41:17 CET LOG:  connection received: host=[local]
2015-03-22 12:41:17 CET LOG:  incomplete startup packet
2015-03-22 12:41:18 CET LOG:  connection received: host=[local]
2015-03-22 12:41:18 CET FATAL:  the database system is starting up
2015-03-22 12:41:18 CET LOG:  restored log file "000000010000000300000065" from archive
2015-03-22 12:41:18 CET LOG:  redo starts at 3/65000028
2015-03-22 12:41:18 CET LOG:  consistent recovery state reached at 3/650000F0
2015-03-22 12:41:18 CET LOG:  connection received: host=[local]
2015-03-22 12:41:18 CET FATAL:  the database system is starting up
2015-03-22 12:41:19 CET LOG:  connection received: host=[local]
2015-03-22 12:41:19 CET FATAL:  the database system is starting up
2015-03-22 12:41:19 CET LOG:  restored log file "000000010000000300000066" from archive
2015-03-22 12:41:19 CET LOG:  recovery stopping after commit of transaction 1213547, time 2015-03-22 12:23:35.834785+01
2015-03-22 12:41:19 CET LOG:  redo done at 3/660000F0
2015-03-22 12:41:19 CET LOG:  last completed transaction was at log time 2015-03-22 12:23:35.834785+01
rsync: link_stat "/var/lib/postgresql/9.3/wal_archive/00000002.history" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [Receiver=3.0.9]
2015-03-22 12:41:19 CET LOG:  selected new timeline ID: 2
rsync: link_stat "/var/lib/postgresql/9.3/wal_archive/00000001.history" failed: No such file or directory (2)
2015-03-22 12:41:19 CET LOG:  connection received: host=[local]
2015-03-22 12:41:19 CET FATAL:  the database system is starting up
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [Receiver=3.0.9]

On remarque que le moteur créée le répertoire des journaux de transaction, ensuite la ligne suivante est assez explicite : “starting point-in-time recovery to XID 1213547″. Quelques lignes plus bas on remarque qu’il rejoue les journaux de transaction (“restored log file “000000010000000300000065” from archive”).

Il arrête la restauration jusqu’à la transaction spécifiée :

recovery stopping after commit of transaction 1213547, time 2015-03-22 12:23:35.834785+01

On remarque quelques lignes d’erreur correspondant à la sortie de rsync. Visiblement il cherche à copier plusieurs fichiers en “x.history”.  En regardant dans le répertoire d’archivage on constate que le moteur a créé un fichier 00000002.history contenant :

1       3/66000120      after transaction 1213547

Point important également, on constate que la numérotation des journaux de transaction a changée :

-rw------- 1 postgres postgres 16777216 mars  22 12:23 000000010000000300000065
-rw------- 1 postgres postgres      305 mars  22 12:23 000000010000000300000065.00000028.backup
-rw------- 1 postgres postgres 16777216 mars  22 12:41 000000010000000300000066
-rw------- 1 postgres postgres 16777216 mars  22 12:46 000000020000000300000066
-rw------- 1 postgres postgres       39 mars  22 12:41 00000002.history

Le fichier “000000010000000300000065” correspond au wal lors du pg_basebackup, “000000010000000300000066” correspond au checkpoint que j’ai lancé après l’incident. Après la restauration on peut voir le fichier “000000020000000300000066”. Le huitième caractère est passé de 1 à 2.

En retournant dans les logs du moteur on peut lire la ligne suivante un peu plus bas : “selected new timeline ID: 2″En fait, lors de la restauration, Postgres créée une nouvelle timeline. C’est un concept que j’aborderai plus tard, en attendant je vous invite à voir cette présentation sur le fonctionnement des timeline après une restauration PIRT : https://wiki.postgresql.org/images/e/e5/FOSDEM2013-Timelines.pdf

Bien entendu comme tout système de sauvegarde vous devez jouer la procédure afin de valider qu’il y a pas eu d’erreur ou oubli.

 

FacebookTwitterGoogle+ViadeoPrintEmailGoogle GmailLinkedInPartager

par Adrien Nayrat le dimanche 22 mars 2015 à 12h25

lundi 16 mars 2015

Damien Clochard

70 Nuances de Postgres

Le support du standard SQL/MED a été introduit dans PostgreSQL 9.1 (2011). Quatre ans plus tard, nous avons maintenant plus de 70 Foreign Data Wrappers (FDW) disponibles pour lire et écrire sur tout type de moteur de stockage: base Oracle ou MongoDB, fichiers XML, cluster Hadoop, dépot Git, Twitter, etc. etc. etc.

Il y a quelque jours, pendant le FOSDEM j’ai assisté à la conférence de mon collègue Ronan Dunklau intitulée Foreign Data Wrappers in PostgreSQL : Where are we now ? Ronan est l’auteur de l’extension multicorn et pendant sa présentation je me suis rappelé à quel point multicorn est probablement le projet le plus sous-estimé de la communauté PostgreSQL. En guise d’exercice, j’ai commencé à compter tous les FDW que je connaissais… et j’ai rapidement réalisé qu’il y en avait trop pour ma mémoire de poisson rouge !

Du coup, je suis retourné sur la liste de FDW du wiki PostgreSQL. J’ai commencé à mettre à jour et reformatter le catalogue de tous les wrappers existants et j’ai abouti à une liste de plus de 70 wrappers pour PostgreSQL…

Quelques leçons à en tirer

Pendant cet inventaire, j’ai découvert quelques points intéressants:

  • Tous les SGBD majeurs sont couverts à l’exception de DB2. Bizarrement DB2 est aussi le seul SGBD majeur ayant une implémentation du standard SQL/MED. Difficile de dire si ces deux faits sont liés ou pas…

  • Un tiers des FDW sont écrits en Python et basés sur Multicorn. Les autres sont des wrappers “natifs” écrits en C. Assez logiquement les wrappers en C sont prvilégiés pour les connecteurs de SGBD (ODBC, Oracle, etc.) pour des raisons de performance. En parallèle Multicorn est utilisé majoritairement pour les interroger des web services (S3 storage,RSS files, etc.) et des formats de données specifiques (fichiers de genotype, GeoJSON, etc.)

  • Je ne pense pas que c’était le but initial mais les Foreign Data Wrappers sont aussi un vecteur d’innovation. Certains connecteurs sont détournés de leur usage premier pour développer de nouvelles fonctionnalités. Par exemple, CitusDB a publié une base de données orientée colonnes basée sur un FDW, d’autres wrappers vont dans des directions encore plus exotiques comme la collecte de stats au niveau OS, de l’accéleration GPU pour les seq scans ou un moteur traitement parallèle de requêtes… La plupart de ces projets ne conviennent pas pour de la production bien sur. Cependant cela démontre qu’il est aisé d’implémenter un PoC et à quel point cette technologie est versatile.

  • Le réseau de distribution PGXN semble plutot méconnu. Seulement 20% des wrappers sont disponibles via PGXN. Ou peut-être que les développeurs sont trop paresseux pour créer un paquet pour leur extension ?

  • Le succès de l’implémentation d’ SQL/MED dans PostgreSQL a toutefois des inconvénients. A vue de nez, il y aura plus de 100 wrappers disponibles à la fin 2015. Pour certaines sources de données comme mongodb ou LDAP, il y a déjà plusieurs wrappers différents, sans parler de la cohorte de connecteurs Hadoop :) Sur le long terme, les utilisateurs de PostgreSQL auront du mal à déterminer quels sont les wrappers maintenus et lesquels sont abandonnés… La page de wiki est une tentative pour répondre à cette question mais il y a surement d’autres solutions pour fournir des informations précises et exhaustives aux utilisateurs. Peut-être qu’il faudrait un data Wrapper qui pourrait lister tous les data wrappers :)

The next big thing : IMPORT FOREIGN SCHEMA

De plus, il reste une limitation majeure à l’implémentation actuelle : les meta données. Actuellement, il n’y a pas de moyen simple d’utiliser les (éventuelles) capacités d’introspection de la source de données distante. En clair : vous devez créer toutes vos tables externes une par une. Lorsque l’on veut se connecter à toutes les tables d’une base distantes, c’est répétitif et il y a un risque d’erreurs. Si vous vous connectez à une base PostgreSQL distante, vous pouvez utiliser des astuces mais clairement c’est une tache qui devrait être faite au niveau du connecteur lui-même.

Heureusement voici la commande IMPORT FOREIGN SCHEMA ! Cette nouvelle fonctionnalité a été écrite par Ronan Dunklau, Michael Paquier et Tom Lane. Vous pouvez lire une démo rapide sur le blog de Michael. Cette commande sera disponible dans la version 9.5.

C’est une amélioration énorme ! Avec cet IMPORT et les douzaines de wrappers disponibles, PostgreSQL est en passe de devenir une plateforme d’intégration de données unique et le besoin d’utiliser des outils d’ETL externes devient de plus en plus faible.

PostgreSQL as a Data Integration Plateform

Liens :

EDIT : Merci à Eric Pommereau et Guénolé Marquier pour la relecture !

lundi 16 mars 2015 à 11h17

jeudi 12 mars 2015

Nicolas Thauvin

pg_back le script de base pour sauvegarder PostgreSQL

Il y a fort longtemps, et c'est ma première contribution relative à PostgreSQL, j'ai écrit un script de backup qui dump tout un serveur PostgreSQL avec pg_dump et pg_dumpall. Il s'agit de pg_back.

Cela peut paraître curieux de publier un simple script de sauvegarde que tout DBA PostgreSQL a écrit dans sa vie et sait écrire par cœur. Surtout qu'on le réécrit en permanence ce script, pour ajuster des chemins, des cas particuliers du serveur à sauvegarder et de l'environnement où l'on sauvegarde...

En bien justement, c'est parce qu'on le réécrit tout le temps que pg_back fait gagner du temps. Il est simple et court, facilement lisible, c'est du shell : tout ce qu'il faut pour en faire une bonne base pour créer un script de sauvegarde adapté. Quand on l'utilise comme patron pour en faire un outil plus évolué, on gagne du temps.

Justement rajouter du code pour l'adapter peut se faire au début. Si on n'a pas envie d'utiliser le fichier de configuration, on adapte la liste de variables au début du script, quitte à en rajouter.

L'autre endroit intéressant c'est tout à la fin, avant le exit, on peut rajouter tout ce qu'il faut pour externaliser ses sauvegardes.

C'est par ici.

jeudi 12 mars 2015 à 15h24

mardi 10 mars 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 8 mars 2015

L'assemblée constituante du futur PostgreSQL Users Group suisse (SwissPUG) aura lieu vendredi 10 avril 2015 : http://www.swisspug.org

Il y aura une session PostgreSQL lors de la conférence sur les BDD (DTCC) le 18 avril 2015 à Beijing : http://dtcc.it168.com/list_jiabin.html

Le pgDay Paris aura lieu le 21 avril 2015 : http://pgday.paris/

Les nouveautés des produits dérivés

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Stephen Frost a poussé :

  • Fix pg_dump handling of extension config tables Since 9.1, we've provided extensions with a way to denote "configuration" tables- tables created by an extension which the user may modify. By marking these as "configuration" tables, the extension is asking for the data in these tables to be pg_dump'd (tables which are not marked in this way are assumed to be entirely handled during CREATE EXTENSION and are not included at all in a pg_dump). Unfortunately, pg_dump neglected to consider foreign key relationships between extension configuration tables and therefore could end up trying to reload the data in an order which would cause FK violations. This patch teaches pg_dump about these dependencies, so that the data dumped out is done so in the best order possible. Note that there's no way to handle circular dependencies, but those have yet to be seen in the wild. The release notes for this should include a caution to users that existing pg_dump-based backups may be invalid due to this issue. The data is all there, but restoring from it will require extracting the data for the configuration tables and then loading them in the correct order by hand. Discussed initially back in bug #6738, more recently brought up by Gilles Darold, who provided an initial patch which was further reworked by Michael Paquier. Further modifications and documentation updates by me. Back-patch to 9.1 where we added the concept of extension configuration tables. http://git.postgresql.org/pg/commitdiff/ebd092bc2a07787b31b249d62033b9c8140a5d85

Robert Haas a poussé :

  • pgbench: Add a real expression syntax to \set. Previously, you could do \set variable operand1 operator operand2, but nothing more complicated. Now, you can \set variable expression, which makes it much simpler to do multi-step calculations here. This also adds support for the modulo operator (%), with the same semantics as in C. Robert Haas and Fabien Coelho, reviewed by Álvaro Herrera and Stephen Frost http://git.postgresql.org/pg/commitdiff/878fdcb843e087cc1cdeadc987d6ef55202ddd04
  • pgbench: Fix mistakes in Makefile. My commit 878fdcb843e087cc1cdeadc987d6ef55202ddd04 was not quite right. Tom Lane pointed out one of the mistakes fixed here, and I noticed the other myself while reviewing what I'd committed. http://git.postgresql.org/pg/commitdiff/e5f36902495d0c8d5dee9a5f43fb45d44540f795
  • Remove residual NULL-pstate handling in addRangeTableEntry. Passing a NULL pstate wouldn't actually work, because isLockedRefname() isn't prepared to cope with it; and there hasn't been any in-core code that tries in over a decade. So just remove the residual NULL handling. Spotted by Coverity; analysis and patch by Michael Paquier. http://git.postgresql.org/pg/commitdiff/5223ddacdc737b401ed58184e321f354bdf46686

Tom Lane a poussé :

  • Fix busted markup. Evidently from commit 878fdcb843e087cc1cdeadc987d6ef55202ddd04. Per buildfarm. http://git.postgresql.org/pg/commitdiff/d1479011744d80d80c669b5bd64dc32187f26c1e
  • Reduce json <=> jsonb casts from explicit-only to assignment level. There's no reason to make users write an explicit cast to store a json value in a jsonb column or vice versa. We could probably even make these implicit, but that might open us up to problems with ambiguous function calls, so for now just do this. http://git.postgresql.org/pg/commitdiff/b67f1ce181910e012b3a8ec7a35ba20a48247757
  • Fix long-obsolete code for separating filter conditions in cost_index(). This code relied on pointer equality to identify which restriction clauses also appear in the indexquals (and, therefore, don't need to be applied as simple filter conditions). That was okay once upon a time, years ago, before we introduced the equivalence-class machinery. Now there's about a 50-50 chance that an equality clause appearing in the indexquals will be the mirror image (commutator) of its mate in the restriction list. When that happens, we'd erroneously think that the clause would be re-evaluated at each visited row, and therefore inflate the cost estimate for the indexscan by the clause's cost. Add some logic to catch this case. It seems to me that it continues not to be worthwhile to expend the extra predicate-proof work that createplan.c will do on the finally-selected plan, but this case is common enough and cheap enough to handle that we should do so. This will make a small difference (about one cpu_operator_cost per row) in simple cases; but in situations where there's an expensive function in the indexquals, it can make a very large difference, as seen in recent example from Jeff Janes. This is a long-standing bug, but I'm hesitant to back-patch because of the possibility of destabilizing plan choices that people may be happy with. http://git.postgresql.org/pg/commitdiff/497bac7d290df13d8b00ba48653a96015ff4741b
  • Fix cost estimation for indexscans on expensive indexed expressions. genericcostestimate() and friends used the cost of the entire indexqual expressions as the charge for initial evaluation of indexscan arguments. But of course the index column is not evaluated, only the other side of the qual expression, so this was a bad overestimate if the index column was an expensive expression. To fix, refactor the logic in this area so that there's a single routine charged with deconstructing index quals and figuring out what is the index column and what is the comparison expression. This is more or less free in the case of btree indexes, since btcostestimate() was doing equivalent deconstruction already. It probably adds a bit of new overhead in the cases of other index types, but not a lot. (In the case of GIN I think I saved something by getting rid of code that wasn't aware that the index column associations were already available "for free".) Per recent gripe from Jeff Janes. Arguably this is a bug fix, but I'm hesitant to back-patch because of the possibility of destabilizing plan choices that people may be happy with. http://git.postgresql.org/pg/commitdiff/b9896198cfbc1b0cd0c631d2af72ffe34bd4c7e5
  • Use standard casting mechanism to convert types in plpgsql, when possible. plpgsql's historical method for converting datatypes during assignments was to apply the source type's output function and then the destination type's input function. Aside from being miserably inefficient in most cases, this method failed outright in many cases where a user might expect it to work; an example is that "declare x int; ... x := 3.9;" would fail, not round the value to 4. Instead, let's convert by applying the appropriate assignment cast whenever there is one. To avoid breaking compatibility unnecessarily, fall back to the I/O conversion method if there is no assignment cast. So far as I can tell, there is just one case where this method produces a different result than the old code in a case where the old code would not have thrown an error. That is assignment of a boolean value to a string variable (type text, varchar, or bpchar); the old way gave boolean's output representation, ie 't'/'f', while the new way follows the behavior of the bool-to-text cast and so gives 'true' or 'false'. This will need to be called out as an incompatibility in the 9.5 release notes. Aside from handling many conversion cases more sanely, this method is often significantly faster than the old way. In part that's because of more effective caching of the conversion info. http://git.postgresql.org/pg/commitdiff/1345cc67bbb014209714af32b5681b1e11eaf964
  • Need to special-case RECORD as well as UNKNOWN in plpgsql's casting logic. This is because can_coerce_type thinks that RECORD can be cast to any composite type, but coerce_record_to_complex only works for inputs that are RowExprs or whole-row Vars, so we get a hard failure on a CaseTestExpr. Perhaps these corner cases ought to be fixed so that coerce_to_target_type actually returns NULL as per its specification, rather than failing ... but for the moment an extra check here is the path of least resistance. http://git.postgresql.org/pg/commitdiff/45f2c2fc4e4adcf75cd689e18dab77ebe622fc2e
  • Change plpgsql's cast cache to consider source typmod as significant. I had thought that there was no need to maintain separate cache entries for different source typmods, but further experimentation shows that there is an advantage to doing so in some cases. In particular, if a domain has a typmod (say, "CREATE DOMAIN d AS numeric(20,0)"), failing to notice the source typmod leads to applying a length-coercion step even when the source has the correct typmod. http://git.postgresql.org/pg/commitdiff/7f3014dce56c7975113809f2ff5e92cf7c1563a3
  • Avoid unused-variable warning in non-assert builds. Oversight in my commit b9896198cfbc1b0cd0c631d2af72ffe34bd4c7e5. http://git.postgresql.org/pg/commitdiff/a5c29d37aab00e9e70e72c97f2be29030f6ee84c
  • Remove comment claiming that PARAM_EXTERN Params always have typmod -1. This hasn't been true in quite some time, cf plpgsql's make_datum_param(). http://git.postgresql.org/pg/commitdiff/3200b15b20d9248be1b0f436ee787b2077d00298
  • Rethink function argument sorting in pg_dump. Commit 7b583b20b1c95acb621c71251150beef958bb603 created an unnecessary dump failure hazard by applying pg_get_function_identity_arguments() to every function in the database, even those that won't get dumped. This could result in snapshot-related problems if concurrent sessions are, for example, creating and dropping temporary functions, as noted by Marko Tiikkaja in bug #12832. While this is by no means pg_dump's only such issue with concurrent DDL, it's unfortunate that we added a new failure mode for cases that used to work, and even more so that the failure was created for basically cosmetic reasons (ie, to sort overloaded functions more deterministically). To fix, revert that patch and instead sort function arguments using information that pg_dump has available anyway, namely the names of the argument types. This will produce a slightly different sort ordering for overloaded functions than the previous coding; but applying strcmp directly to the output of pg_get_function_identity_arguments really was a bit odd anyway. The sorting will still be name-based and hence independent of possibly-installation-specific OID assignments. A small additional benefit is that sorting now works regardless of server version. Back-patch to 9.3, where the previous commit appeared. http://git.postgresql.org/pg/commitdiff/e3bfe6d84d4919433d8323cfb8194ca60d99f2c4
  • Fix erroneous error message for REINDEX SYSTEM. Missed case in commit fe263d115a7dd16095b8b8f1e943aff2bb4574d2. Sawada Masahiko http://git.postgresql.org/pg/commitdiff/ac0914285ac90bd411730c3219f226bbbbc57f3a
  • Code cleanup for REINDEX DATABASE/SCHEMA/SYSTEM. Fix some minor infelicities. Some of these things were introduced in commit fe263d115a7dd16095b8b8f1e943aff2bb4574d2, and some are older. http://git.postgresql.org/pg/commitdiff/90c35a9ed06c1353a0d3818c259e629ff09dba18
  • Fix documentation for libpq's PQfn(). The SGML docs claimed that 1-byte integers could be sent or received with the "isint" options, but no such behavior has ever been implemented in pqGetInt() or pqPutInt(). The in-code documentation header for PQfn() was even less in tune with reality, and the code itself used parameter names matching neither the SGML docs nor its libpq-fe.h declaration. Do a bit of additional wordsmithing on the SGML docs while at it. Since the business about 1-byte integers is a clear documentation bug, back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/1a0bc4c2bfc278b63965486b1525ad04a1f85989
  • Remove struct PQArgBlock from server-side header libpq/libpq.h. This struct is purely a client-side artifact. Perhaps there was once reason for the server to know it, but any such reason is lost in the mists of time. We certainly don't need two independent declarations of it. http://git.postgresql.org/pg/commitdiff/01cca2c1b1a0d52c83f250c50942ee00e62637ca
  • Cast to (void *) rather than (int *) when passing int64's to PQfn(). This is a possibly-vain effort to silence a Coverity warning about bogus endianness dependency. The code's fine, because it takes care of endianness issues for itself, but Coverity sees an int64 being passed to an int* argument and not unreasonably suspects something's wrong. I'm not sure if putting the void* cast in the way will shut it up; but it can't hurt and seems better from a documentation standpoint anyway, since the pointer is not used as an int* in this code path. Just for a bit of additional safety, verify that the result length is 8 bytes as expected. Back-patch to 9.3 where the code in question was added. http://git.postgresql.org/pg/commitdiff/ef75508efc789c79c5a5d4acd7ad5da85f1e4f08

Álvaro Herrera a poussé :

  • Add comment for "is_internal" parameter. This was missed in my commit f4c4335 of 9.3 vintage, so backpatch to that. http://git.postgresql.org/pg/commitdiff/6f9d79904748c26a58991942dc6719db558f77b0
  • Change many routines to return ObjectAddress rather than OID. The changed routines are mostly those that can be directly called by ProcessUtilitySlow; the intention is to make the affected object information more precise, in support for future event trigger changes. Originally it was envisioned that the OID of the affected object would be enough, and in most cases that is correct, but upon actually implementing the event trigger changes it turned out that ObjectAddress is more widely useful. Additionally, some command execution routines grew an output argument that's an object address which provides further info about the executed command. To wit: * for ALTER DOMAIN / ADD CONSTRAINT, it corresponds to the address of the new constraint * for ALTER OBJECT / SET SCHEMA, it corresponds to the address of the schema that originally contained the object. * for ALTER EXTENSION {ADD, DROP} OBJECT, it corresponds to the address of the object added to or dropped from the extension. There's no user-visible change in this commit, and no functional change either. Discussion: 20150218213255.GC6717@tamriel.snowman.net Reviewed-By: Stephen Frost, Andres Freund http://git.postgresql.org/pg/commitdiff/a2e35b53c39b2a27d3e332dc7c506539c306fd44
  • Silence warning in non-assert-enabled build. An OID return value was being used only for a (rather pointless) assert. Silence by removing the variable and the assert. Per note from Peter Geoghegan http://git.postgresql.org/pg/commitdiff/bf22d2707a2f47a7cc4caa239a14f2bf0a72bfd0
  • Fix user mapping object description. We were using "user mapping for user XYZ" as description for user mappings, but that's ambiguous because users can have mappings on multiple foreign servers; therefore change it to "for user XYZ on server UVW" instead. Object identities for user mappings are also updated in the same way, in branches 9.3 and above. The incomplete description string was introduced together with the whole SQL/MED infrastructure by commit cae565e503 of 8.4 era, so backpatch all the way back. http://git.postgresql.org/pg/commitdiff/cf34e373fcf42239a73f36e3054d9e9fbdc1e0de
  • Fix contrib/file_fdw's expected file. I forgot to update it on yesterday's cf34e373fcf. http://git.postgresql.org/pg/commitdiff/c6ee39bc8587042f018979ddd6ed9825acbbd3d8
  • Add some more tests on event triggers. Fabien Coelho. Reviewed by Robert Haas http://git.postgresql.org/pg/commitdiff/6510c832bbf91d52541c7aeefa371123abc2d832

Fujii Masao a poussé :

Peter Eisentraut a poussé :

Noah Misch a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in a patch to fix broken Install.bat when target directory contains a space.
  • Jan de Visser sent in two more revisions of a patch to let pg ctl check the result of a postmaster config reload.
  • Gregory Stark and Haribabu Kommi traded patches to provide a catalog view of pg_hba.conf.
  • Joshua Berkus sent in four revisions of a patch to raise default for max_wal_segments to 1GB.
  • SAWADA Masahiko sent in another revision of a patch to implement REINDEX...VERBOSE.
  • Dean Rasheed sent in two more revisions of a patch to update RLS timings.
  • Álvaro Herrera sent in four more revisions of a patch to implement ALTER USER/ROLE ... CURRENT USER.
  • Michael Paquier sent in two more revisions of a patch to improve test coverage with pg_dump.
  • Shigeru HANADA and Ashutosh Bapat traded patches to implement push-down JOIN support for foreign tables.
  • Marko Kreen sent in a patch to fix excessive float lossiness in PL/Python.
  • Julien Tachoires sent in another revision of a patch to allow toast tables to be moved to a different tablespace.
  • Amit Kapila sent in another revision of a patch to implement parallel seq scan.
  • Kaigai Kouhei and Shigeru HANADA traded patches to add custom foreign join APIs.
  • Peter Geoghegan sent in another revision of a patch to add logical decoding support for ON CONFLICT UPDATE.
  • Peter Geoghegan sent in a patch to remove an obsolete SnapshotNow reference within snapbuild.c.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add regrole and regnamespace.
  • Michael Paquier sent in a patch to compare primary/HS standby in tests.
  • Robert Haas sent in a patch to make some fixes in tuplesort.
  • Rahila Syed sent in two more revisions of a patch to allow compressing improve full-page writes.
  • Fabien COELHO sent in four more revisions of a patch to improve pgbench syntax error messages.
  • David Rowley sent in another revision of a patch to improve performance for joins where outer side is unique.
  • Michael Paquier sent in a patch to install shared libraries in bin/ and lib/ on MSVC.
  • Michael Paquier sent in a patch to add dummy pstate fixes.
  • Peter Geoghegan sent in another revision of a patch to add INSERT ... ON CONFLICT IGNORE (and UPDATE).
  • Andreas Karlsson sent in another revision of a patch to use 128-bit integers for sum, avg and statistics aggregates.
  • Etsuro Fujita sent in another revision of a patch to make updating foreign tables in the Postgres FDW work faster.
  • SAWADA Masahiko sent in another revision of a patch to add a way to see the contents of configuration files via SQL.
  • Michael Paquier sent in two more revisions of a patch to add a table level log_autovacuum_min_duration.
  • Michael Paquier sent in a flock of patches to move the freeze parameters of VacuumStmt into a separate spot, eliminate VacuumStmt from lower level routines of ANALYZE and VACUUM, and add wraparound control parameters in VacuumStmt.
  • Bruce Momjian sent in two more revisions of a patch to help fix pg_upgrade with reference to rsync.
  • Robert Haas sent in another revision of a patch to implement parallel mode and parallel contexts.
  • Tom Lane sent in a patch to fix some weirdly pesimistic estimates in optimizer.
  • Álvaro Herrera sent in a flock of patches to patches add get_object_address support for user mappings, default ACLs, and operators and functions of operator families
  • Kyotaro HORIGUCHI and Tom Lane traded patches to clamp row number of join product by the row number calculated from joining paths.
  • Stephen Frost sent in another revision of a patch to add catalog_function_acls.
  • Marco Nenciarini sent in another revision of a patch to implement file-based incremental backup.
  • SAWADA Masahiko sent in a patch to fix an incorrect error message in REINDEX.
  • Fabrízio de Royes Mello sent in two revisions of a patch to fix an odd debug in walreceiver.
  • Peter Eisentraut sent in another revision of a patch to speed up make check-world.
  • Peter Eisentraut sent in another revision of a patch to add TRANSFORMS.
  • Pavel Stehule sent in a PoC patch to enforce casting to most common type automatically.
  • Dmitry Voronin sent in a patch to adds functions to sslinfo extension module: ssl_extension_names(), ssl_extension_value(text), and ssl_extension_is_critical(text).
  • Tomas Vondra sent in a patch to allow merging pgbench logs.
  • Tomas Vondra sent in a patch to allow logging both aggregate and transaction info in pgbench, rather having to choose one or the other.
  • Tom Lane sent in a patch to rethink the parameter access hooks for plpgsql's benefit.

par N Bougain le mardi 10 mars 2015 à 22h42

jeudi 5 mars 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 1er mars 2015

Le PUG de Manchester (Royaume-Uni) redémarre le 4 mars. Détails et RSVP ci-après : http://www.meetup.com/The-Manchester-PostgreSQL-Meetup/

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Heikki Linnakangas a poussé :

  • Fix potential deadlock with libpq non-blocking mode. If libpq output buffer is full, pqSendSome() function tries to drain any incoming data. This avoids deadlock, if the server e.g. sends a lot of NOTICE messages, and blocks until we read them. However, pqSendSome() only did that in blocking mode. In non-blocking mode, the deadlock could still happen. To fix, take a two-pronged approach: 1. Change the documentation to instruct that when PQflush() returns 1, you should wait for both read- and write-ready, and call PQconsumeInput() if it becomes read-ready. That fixes the deadlock, but applications are not going to change overnight. 2. In pqSendSome(), drain the input buffer before returning 1. This alleviates the problem for applications that only wait for write-ready. In particular, a slow but steady stream of NOTICE messages during COPY FROM STDIN will no longer cause a deadlock. The risk remains that the server attempts to send a large burst of data and fills its output buffer, and at the same time the client also sends enough data to fill its output buffer. The application will deadlock if it goes to sleep, waiting for the socket to become write-ready, before the server's data arrives. In practice, NOTICE messages and such that the server might be sending are usually short, so it's highly unlikely that the server would fill its output buffer so quickly. Backpatch to all supported versions. http://git.postgresql.org/pg/commitdiff/2a3f6e368babdac7b586a7d43105af60fc08b1a3
  • Refactor unit conversions code in guc.c. Replace the if-switch-case constructs with two conversion tables, containing all the supported conversions between human-readable unit strings and the base units used in GUC variables. This makes the code easier to read, and makes adding new units simpler. http://git.postgresql.org/pg/commitdiff/1b6302647302f459fdb63008c3842a1b0d43d1b7
  • Renumber GUC_* constants. This moves all the regular flags back together (for aesthetic reasons), and makes room for more GUC_UNIT_* types. http://git.postgresql.org/pg/commitdiff/0fec000365c25fd89ea583673de226e816dba60f
  • Replace checkpoint_segments with min_wal_size and max_wal_size. Instead of having a single knob (checkpoint_segments) that both triggers checkpoints, and determines how many checkpoints to recycle, they are now separate concerns. There is still an internal variable called CheckpointSegments, which triggers checkpoints. But it no longer determines how many segments to recycle at a checkpoint. That is now auto-tuned by keeping a moving average of the distance between checkpoints (in bytes), and trying to keep that many segments in reserve. The advantage of this is that you can set max_wal_size very high, but the system won't actually consume that much space if there isn't any need for it. The min_wal_size sets a floor for that; you can effectively disable the auto-tuning behavior by setting min_wal_size equal to max_wal_size. The max_wal_size setting is now the actual target size of WAL at which a new checkpoint is triggered, instead of the distance between checkpoints. Previously, you could calculate the actual WAL usage with the formula "(2 + checkpoint_completion_target) * checkpoint_segments + 1". With this patch, you set the desired WAL usage with max_wal_size, and the system calculates the appropriate CheckpointSegments with the reverse of that formula. That's a lot more intuitive for administrators to set. Reviewed by Amit Kapila and Venkata Balaji N. http://git.postgresql.org/pg/commitdiff/88e982302684246e8af785e78a467ac37c76dee9
  • Fix typo in README. Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/dd58c6098f2f2fcea71c7125f9594268a24a38ad
  • Fix recovery_command -> restore_command typo in 8.3 release notes. Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/347c74320d10bee458d1fec159aeda7143d31bfb

Fujii Masao a poussé :

  • Add GUC to control the time to wait before retrieving WAL after failed attempt. Previously when the standby server failed to retrieve WAL files from any sources (i.e., streaming replication, local pg_xlog directory or WAL archive), it always waited for five seconds (hard-coded) before the next attempt. For example, this is problematic in warm-standby because restore_command can fail every five seconds even while new WAL file is expected to be unavailable for a long time and flood the log files with its error messages. This commit adds new parameter, wal_retrieve_retry_interval, to control that wait time. Alexey Vasiliev and Michael Paquier, reviewed by Andres Freund and me. http://git.postgresql.org/pg/commitdiff/5d2b45e3f78a85639f30431181c06d4c3221c5a1
  • Add note about how to make the SRF detoasted arguments live accross calls. Andrew Gierth and Ali Akbar http://git.postgresql.org/pg/commitdiff/a7920b872fff36668a2d33157609024b851b5c2e

Andres Freund a poussé :

  • Guard against spurious signals in LockBufferForCleanup. When LockBufferForCleanup() has to wait for getting a cleanup lock on a buffer it does so by setting a flag in the buffer header and then wait for other backends to signal it using ProcWaitForSignal(). Unfortunately LockBufferForCleanup() missed that ProcWaitForSignal() can return for other reasons than the signal it is hoping for. If such a spurious signal arrives the wait flags on the buffer header will still be set. That then triggers "ERROR: multiple backends attempting to wait for pincount 1". The fix is simple, unset the flag if still set when retrying. That implies an additional spinlock acquisition/release, but that's unlikely to matter given the cost of waiting for a cleanup lock. Alternatively it'd have been possible to move responsibility for maintaining the relevant flag to the waiter all together, but that might have had negative consequences due to possible floods of signals. Besides being more invasive. This looks to be a very longstanding bug. The relevant code in LockBufferForCleanup() hasn't changed materially since its introduction and ProcWaitForSignal() was documented to return for unrelated reasons since 8.2. The master only patch series removing ImmediateInterruptOK made it much easier to hit though, as ProcSendSignal/ProcWaitForSignal now uses a latch shared with other tasks. Per discussion with Kevin Grittner, Tom Lane and me. Backpatch to all supported branches. Discussion: 11553.1423805224@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/bc208a5a2f9eb34bf795374fa5381e0c82073255
  • Reconsider when to wait for WAL flushes/syncrep during commit. Up to now RecordTransactionCommit() waited for WAL to be flushed (if synchronous_commit != off) and to be synchronously replicated (if enabled), even if a transaction did not have a xid assigned. The primary reason for that is that sequence's nextval() did not assign a xid, but are worthwhile to wait for on commit. This can be problematic because sometimes read only transactions do write WAL, e.g. HOT page prune records. That then could lead to read only transactions having to wait during commit. Not something people expect in a read only transaction. This lead to such strange symptoms as backends being seemingly stuck during connection establishment when all synchronous replicas are down. Especially annoying when said stuck connection is the standby trying to reconnect to allow syncrep again... This behavior also is involved in a rather complicated <= 9.4 bug where the transaction started by catchup interrupt processing waited for syncrep using latches, but didn't get the wakeup because it was already running inside the same overloaded signal handler. Fix the issue here doesn't properly solve that issue, merely papers over the problems. In 9.5 catchup interrupts aren't processed out of signal handlers anymore. To fix all this, make nextval() acquire a top level xid, and only wait for transaction commit if a transaction both acquired a xid and emitted WAL records. If only a xid has been assigned we don't uselessly want to wait just because of writes to temporary/unlogged tables; if only WAL has been written we don't want to wait just because of HOT prunes. The xid assignment in nextval() is unlikely to cause overhead in real-world workloads. For one it only happens SEQ_LOG_VALS/32 values anyway, for another only usage of nextval() without using the result in an insert or similar is affected. Discussion: 20150223165359.GF30784@awork2.anarazel.de, 369698E947874884A77849D8FE3680C2@maumau, 5CF4ABBA67674088B3941894E22A0D25@maumau Per complaint from maumau and Thom Brown Backpatch all the way back; 9.0 doesn't have syncrep, but it seems better to be consistent behavior across all maintained branches. http://git.postgresql.org/pg/commitdiff/fd6a3f3ad4067f1b8fc28e9de6e99e5936d82161

Tom Lane a poussé :

  • Further tweaking of raw grammar output to distinguish different inputs. Use a different A_Expr_Kind for LIKE/ILIKE/SIMILAR TO constructs, so that they can be distinguished from direct invocation of the underlying operators. Also, postpone selection of the operator name when transforming "x IN (select)" to "x = ANY (select)", so that those syntaxes can be told apart at parse analysis time. I had originally thought I'd also have to do something special for the syntaxes IS NOT DISTINCT FROM, IS NOT DOCUMENT, and x NOT IN (SELECT...), which the grammar translates as though they were NOT (construct). On reflection though, we can distinguish those cases reliably by noting whether the parse location shown for the NOT is the same as for its child node. This only requires tweaking the parse locations for NOT IN, which I've done here. These changes should have no effect outside the parser; they're just in support of being able to give accurate warnings for planned operator precedence changes. http://git.postgresql.org/pg/commitdiff/56be925e4b8f5b1c0e6716ca5cbe0360d1229f50
  • Improve parser's one-extra-token lookahead mechanism. There are a couple of places in our grammar that fail to be strict LALR(1), by requiring more than a single token of lookahead to decide what to do. Up to now we've dealt with that by using a filter between the lexer and parser that merges adjacent tokens into one in the places where two tokens of lookahead are necessary. But that creates a number of user-visible anomalies, for instance that you can't name a CTE "ordinality" because "WITH ordinality AS ..." triggers folding of WITH and ORDINALITY into one token. I realized that there's a better way. In this patch, we still do the lookahead basically as before, but we never merge the second token into the first; we replace just the first token by a special lookahead symbol when one of the lookahead pairs is seen. This requires a couple extra productions in the grammar, but it involves fewer special tokens, so that the grammar tables come out a bit smaller than before. The filter logic is no slower than before, perhaps a bit faster. I also fixed the filter logic so that when backing up after a lookahead, the current token's terminator is correctly restored; this eliminates some weird behavior in error message issuance, as is shown by the one change in existing regression test outputs. I believe that this patch entirely eliminates odd behaviors caused by lookahead for WITH. It doesn't really improve the situation for NULLS followed by FIRST/LAST unfortunately: those sequences still act like a reserved word, even though there are cases where they should be seen as two ordinary identifiers, eg "SELECT nulls first FROM ...". I experimented with additional grammar hacks but couldn't find any simple solution for that. Still, this is better than before, and it seems much more likely that we *could* somehow solve the NULLS case on the basis of this filter behavior than the previous one. http://git.postgresql.org/pg/commitdiff/d809fd0008a2e26de463f47b7aba0365264078f3
  • Fix dumping of views that are just VALUES(...) but have column aliases. The "simple" path for printing VALUES clauses doesn't work if we need to attach nondefault column aliases, because there's noplace to do that in the minimal VALUES() syntax. So modify get_simple_values_rte() to detect nondefault aliases and treat that as a non-simple case. This further exposes that the "non-simple" path never actually worked; it didn't produce valid syntax. Fix that too. Per bug #12789 from Curtis McEnroe, and analysis by Andrew Gierth. Back-patch to all supported branches. Before 9.3, this also requires back-patching the part of commit 092d7ded29f36b0539046b23b81b9f0bf2d637f1 that created get_simple_values_rte() to begin with; inserting the extra test into the old factorization of that logic would've been too messy. http://git.postgresql.org/pg/commitdiff/e9f1c01b71dcd11c86fc8516c06dae2e784b96fd
  • Fix over-optimistic caching in fetch_array_arg_replace_nulls(). When I rewrote this in commit 56a79a869bedc4bf6c35853642694cc0b0594dd2, I forgot that it's possible for the input array type to change from one call to the next (this can happen when applying the function to pg_statistic columns, for instance). Fix that. http://git.postgresql.org/pg/commitdiff/77903ede08845e55bd2a6c99b52d8da6926d6e84
  • Redefine MemoryContextReset() as deleting, not resetting, child contexts. That is, MemoryContextReset() now means what was formerly meant by MemoryContextResetAndDeleteChildren(), and the latter is now just a macro alias for the former. If you really want the functionality that was formerly provided by MemoryContextReset(), what you have to do is MemoryContextResetChildren() plus MemoryContextResetOnly() (which is a new API to reset *only* the named context and not touch its children). The reason for this change is that near fifteen years of experience has proven that there is noplace where old-style MemoryContextReset() is actually what you want. Making that the default behavior has led to lots of context-leakage bugs, while we've not found anyplace where it's actually necessary to keep the child contexts; at least the standard regression tests do not reveal anyplace where this change breaks anything. And there are upcoming patches that will introduce additional reasons why child contexts need to be removed. We could change existing calls of MemoryContextResetAndDeleteChildren to be just MemoryContextReset, but for the moment I'll leave them alone; they're not costing anything. http://git.postgresql.org/pg/commitdiff/eaa5808e8ec4e82ce1a87103a6b6f687666e4e4c
  • Suppress uninitialized-variable warning from less-bright compilers. The type variable must get set on first iteration of the while loop, but there are reasonably modern gcc versions that don't realize that. Initialize it with a dummy value. This undoes a removal of initialization in commit 654809e770ce270c0bb9de726c5df1ab193d60f0. http://git.postgresql.org/pg/commitdiff/d61f1a93271b859938b6cdb301a367cdb8abf81c
  • Improve mmgr README. Add documentation about the new reset callback mechanism. Also, at long last, recast the existing text so that it describes the current context mechanisms as established fact rather than something we're going to implement. Shoulda done that in 2001 or so ... http://git.postgresql.org/pg/commitdiff/c4f4c7ca99169ac609df67c2d0eb78162484420c
  • Fix planning of star-schema-style queries. Part of the intent of the parameterized-path mechanism was to handle star-schema queries efficiently, but some overly-restrictive search limiting logic added in commit e2fa76d80ba571d4de8992de6386536867250474 prevented such cases from working as desired. Fix that and add a regression test about it. Per gripe from Marc Cousin. This is arguably a bug rather than a new feature, so back-patch to 9.2 where parameterized paths were introduced. http://git.postgresql.org/pg/commitdiff/b514a7460d9127ddda6598307272c701cbb133b7
  • Track typmods in plpgsql expression evaluation and assignment. The main value of this change is to avoid expensive I/O conversions when assigning to a variable that has a typmod specification, if the value to be assigned is already known to have the right typmod. This is particularly valuable for arrays with typmod specifications; formerly, in an assignment to an array element the entire array would invariably get put through double I/O conversion to check the typmod, to absolutely no purpose since we'd already properly coerced the new element value. Extracted from my "expanded arrays" patch; this seems worth committing separately, whatever becomes of that patch, since it's really an independent issue. As long as we're changing the function signatures, take the opportunity to rationalize the argument lists of exec_assign_value, exec_cast_value, and exec_simple_cast_value; that is, put the arguments into a saner order, and get rid of the bizarre choice to pass exec_assign_value's isNull flag by reference. http://git.postgresql.org/pg/commitdiff/e524cbdc45ec6d677b1dd49ee64dd403959eda0f
  • Invent a memory context reset/delete callback mechanism. This allows cleanup actions to be registered to be called just before a particular memory context's contents are flushed (either by deletion or MemoryContextReset). The patch in itself has no use-cases for this, but several likely reasons for wanting this exist. In passing, per discussion, rearrange some boolean fields in struct MemoryContextData so as to avoid wasted padding space. For safety, this requires making allowInCritSection's existence unconditional; but I think that's a better approach than what was there anyway. http://git.postgresql.org/pg/commitdiff/f65e8270587f3e9b8224e20f7d020ed1f816dfe1
  • Move memory context callback declarations into palloc.h. Initial experience with this feature suggests that instances of MemoryContextCallback are likely to propagate into some widely-used headers over time. As things stood, that would result in pulling memutils.h or at least memnodes.h into common headers, which does not seem desirable. Instead, let's decide that this feature is part of the "ordinary palloc user" API rather than the "specialized context management" API, and as such should be declared in palloc.h not memutils.h. http://git.postgresql.org/pg/commitdiff/097fe194aa7c590b4fa43d5e40c083940859c286
  • Use the typcache to cache constraints for domain types. Previously, we cached domain constraints for the life of a query, or really for the life of the FmgrInfo struct that was used to invoke domain_in() or domain_check(). But plpgsql (and probably other places) are set up to cache such FmgrInfos for the whole lifespan of a session, which meant they could be enforcing really stale sets of constraints. On the other hand, searching pg_constraint once per query gets kind of expensive too: testing says that as much as half the runtime of a trivial query such as "SELECT 0::domaintype" went into that. To fix this, delegate the responsibility for tracking a domain's constraints to the typcache, which has the infrastructure needed to detect syscache invalidation events that signal possible changes. This not only removes unnecessary repeat reads of pg_constraint, but ensures that we never apply stale constraint data: whatever we use is the current data according to syscache rules. Unfortunately, the current configuration of the system catalogs means we have to flush cached domain-constraint data whenever either pg_type or pg_constraint changes, which happens rather a lot (eg, creation or deletion of a temp table will do it). It might be worth rearranging things to split pg_constraint into two catalogs, of which the domain constraint one would probably be very low-traffic. That's a job for another patch though, and in any case this patch should improve matters materially even with that handicap. This patch makes use of the recently-added memory context reset callback feature to manage the lifespan of domain constraint caches, so that we don't risk deleting a cache that might be in the midst of evaluation. Although this is a bug fix as well as a performance improvement, no back-patch. There haven't been many if any field complaints about stale domain constraint checks, so it doesn't seem worth taking the risk of modifying data structures as basic as MemoryContexts in back branches. http://git.postgresql.org/pg/commitdiff/8abb3cda0ddc00a0ab98977a1633a95b97068d4e

Álvaro Herrera a poussé :

  • Fix stupid merge errors in previous commit. Brown paper bag installed permanently. http://git.postgresql.org/pg/commitdiff/d1712d01d01d2c14355fad497fa7a6ae6e33694f
  • Support more commands in event triggers. COMMENT, SECURITY LABEL, and GRANT/REVOKE now also fire ddl_command_start and ddl_command_end event triggers, when they operate on database-local objects. Reviewed-By: Michael Paquier, Andres Freund, Stephen Frost http://git.postgresql.org/pg/commitdiff/296f3a6053844089bc533630fffafaba8f016384
  • Fix table_rewrite event trigger for ALTER TYPE/SET DATA TYPE CASCADE. When a composite type being used in a typed table is modified by way of ALTER TYPE, a table rewrite occurs appearing to come from ALTER TYPE. The existing event_trigger.c code was unable to cope with that and raised a spurious error. The fix is just to accept that command tag for the event, and document this properly. Noted while fooling with deparsing of DDL commands. This appears to be an oversight in commit 618c9430a. Thanks to Mark Wong for documentation wording help. http://git.postgresql.org/pg/commitdiff/3f190f67eb45ae61d696fbce8ab48d246a12f709
  • Make CREATE OR REPLACE VIEW internally more consistent. The way that columns are added to a view is by calling AlterTableInternal with special subtype AT_AddColumnToView; but that subtype is changed to AT_AddColumnRecurse by ATPrepAddColumn. This has no visible effect in the current code, since views cannot have inheritance children (thus the recursion step is a no-op) and adding a column to a view is executed identically to doing it to a table; but it does make a difference for future event trigger code keeping track of commands, because the current situation leads to confusing the case with a normal ALTER TABLE ADD COLUMN. Fix the problem by passing a flag to ATPrepAddColumn to prevent it from changing the command subtype. The event trigger code can then properly ignore the subcommand. (We could remove the call to ATPrepAddColumn, since views are never typed, and there is never a need for recursion, which are the two conditions that are checked by ATPrepAddColumn; but it seems more future-proof to keep the call in place.) http://git.postgresql.org/pg/commitdiff/fbef4342a86522c98cd605891ad8c1145a61d191
  • Fix a couple of trivial issues in jsonb.c. Typo "aggreagate" appeared three times, and the return value of function JsonbIteratorNext() was being assigned to an int variable in a bunch of places. http://git.postgresql.org/pg/commitdiff/654809e770ce270c0bb9de726c5df1ab193d60f0
  • Fix intermittent failure in event_trigger test. As evidenced by measles in buildfarm. Pointed out by Tom Lane. http://git.postgresql.org/pg/commitdiff/e059e02e43cd825616192d010e9e638a96ad4717

Peter Eisentraut a poussé :

  • Fix invalid DocBook XML http://git.postgresql.org/pg/commitdiff/b007bee1f6962ad1007056f64b9eb2e505fa6806
  • Error when creating names too long for tar format The tar format (at least the version we are using), does not support file names or symlink targets longer than 99 bytes. Until now, the tar creation code would silently truncate any names that are too long. (Its original application was pg_dump, where this never happens.) This creates problems when running base backups over the replication protocol. The most important problem is when a tablespace path is longer than 99 bytes, which will result in a truncated tablespace path being backed up. Less importantly, the basebackup protocol also promises to back up any other files it happens to find in the data directory, which would also lead to file name truncation if someone put a file with a long name in there. Now both of these cases result in an error during the backup. Add tests that fail when a too-long file name or symlink is attempted to be backed up. Reviewed-by: Robert Haas <robertmhaas@gmail.com> http://git.postgresql.org/pg/commitdiff/23a78352c0a0dc21d6120bd868f0b2d07395b537

Michael Meskes a poussé :

Stephen Frost a poussé :

  • Add locking clause for SB views for update/delete. In expand_security_qual(), we were handling locking correctly when a PlanRowMark existed, but not when we were working with the target relation (which doesn't have any PlanRowMarks, but the subquery created for the security barrier quals still needs to lock the rows under it). Noted by Etsuro Fujita when working with the Postgres FDW, which wasn't properly issuing a SELECT ... FOR UPDATE to the remote side under a DELETE. Back-patch to 9.4 where updatable security barrier views were introduced. Per discussion with Etsuro Fujita and Dean Rasheed. http://git.postgresql.org/pg/commitdiff/6f9bd50eabb0a4960e94c83dac8855771c9f340d
  • Add hasRowSecurity to copyfuncs/outfuncs. The RLS patch added a hasRowSecurity field to PlannerGlobal and PlannedStmt but didn't update nodes/copyfuncs.c and nodes/outfuncs.c to reflect those additional fields. Correct that by adding entries to the appropriate functions for those fields. Pointed out by Robert Haas. http://git.postgresql.org/pg/commitdiff/62a4a1af5d1d84f0023bc3816c204191b0f4f49f
  • Fix targetRelation initializiation in prepsecurity. In 6f9bd50eabb0a4960e94c83dac8855771c9f340d, we modified expand_security_quals() to tell expand_security_qual() about when the current RTE was the targetRelation. Unfortunately, that commit initialized the targetRelation variable used outside of the loop over the RTEs instead of at the start of it. This patch moves the variable and the initialization of it into the loop, where it should have been to begin with. Pointed out by Dean Rasheed. Back-patch to 9.4 as the original commit was. http://git.postgresql.org/pg/commitdiff/ee4ddcb38a0abfdb8f7302bbc332a1cb92888ed1

Noah Misch a poussé :

  • Free SQLSTATE and SQLERRM no earlier than other PL/pgSQL variables. "RETURN SQLERRM" prompted plpgsql_exec_function() to read from freed memory. Back-patch to 9.0 (all supported versions). Little code ran between the premature free and the read, so non-assert builds are unlikely to witness user-visible consequences. http://git.postgresql.org/pg/commitdiff/f5ef00aed4c39645716cabb2e4cf1ef3598fcde7
  • Unlink static libraries before rebuilding them. When the library already exists in the build directory, "ar" preserves members not named on its command line. This mattered when, for example, a "configure" rerun dropped a file from $(LIBOBJS). libpgport carried the obsolete member until "make clean". Back-patch to 9.0 (all supported versions). http://git.postgresql.org/pg/commitdiff/424793fa5dc631254f69d5ee8d7d7d6de2976f60
  • Add transform functions for AT TIME ZONE. This makes "ALTER TABLE tabname ALTER tscol TYPE ... USING tscol AT TIME ZONE 'UTC'" skip rewriting the table when altering from "timestamp" to "timestamptz" or vice versa. While it would be nicer still to optimize this in the absence of the USING clause given timezone==UTC, transform functions must consult IMMUTABLE facts only. http://git.postgresql.org/pg/commitdiff/b8a18ad4850ea5ad7884aa6ab731fd392e73b4ad

Andrew Dunstan a poussé :

  • Render infinite date/timestamps as 'infinity' for json/jsonb. Commit ab14a73a6c raised an error in these cases and later the behaviour was copied to jsonb. This is what the XML code, which we then adopted, does, as the XSD types don't accept infinite values. However, json dates and timestamps are just strings as far as json is concerned, so there is no reason not to render these values as 'infinity'. The json portion of this is backpatched to 9.4 where the behaviour was introduced. The jsonb portion only affects the development branch. Per gripe on pgsql-general. http://git.postgresql.org/pg/commitdiff/bda76c1c8cfb1d11751ba6be88f0242850481733

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Corey Huinker sent in another revision of a patch to add polymorphic functions to dblink.
  • Tom Lane sent in two more revisions of a patch to make operator precedent standards compliant and sane.
  • Michael Paquier sent in two more revisions of a patch to enforce creation of input and output paths in pg_regress, and add regress_dynamic as a test module.
  • David Steele sent in two more revisions of a patch to add pg_audit as an extension.
  • Álvaro Herrera sent in a patch to remove the OBJECT_ATTRIBUTE symbol.
  • Peter Eisentraut sent in another revision of a patch to add logical decoding support for ON CONFLICT UPDATE.
  • Andres Freund sent in a patch to reconsider when to flush WAL and wait for syncrep while committing.
  • Rahila Syed sent in three more revisions of a patch to compress full-page writes.
  • Álvaro Herrera and David Fetter traded patches to fix a bug in psql's handling of connect strings and URIs in psql's \connect.
  • Tomas Vondra sent in another revision of a patch to separate logical from physical column ordering.
  • Michael Paquier and Stephen Frost traded patches to fix a dependency issue in pg_dump which mis-ordered tables in extensions.
  • Kyotaro HORIGUCHI sent in another revision of a patch to ensure that pg_basebackup sends feedback regularly, even under heavy loads.
  • Tom Lane sent in a patch to ensure that NOT LIKE, etc., have reasonable precedence.
  • Andreas Karlsson sent in a patch to fix some autoconf deprecation warnings.
  • Amit Langote sent in two revisions of a patch to implement native partitioning.
  • Álvaro Herrera sent in two flocks of patches for deparsing utility commands.
  • Alexander Korotkov sent in another revision of a patch to add KNN-GiST with recheck.
  • Michael Paquier sent in a patch to add NULL-pointer check and incorrect comment for pstate in addRangeTableEntry.
  • Fabien COELHO sent in two more revisions of a patch to add more tests on event triggers.
  • Tom Lane sent in another revision of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Fabrízio de Royes Mello sent in a patch to add CINE for ALTER TABLE ... ADD COLUMN.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add new OID alias type regrole.
  • Tom Lane sent in a patch to make MemoryContextReset delete children.
  • Tom Lane sent in a patch to add MemoryContext reset/delete callbacks.
  • Haribabu Kommi sent in another revision of a patch to provide a catalog view to the pg_hba.conf file.
  • Kyotaro HORIGUCHI and Kevin Grittner traded patches to reduce pinning in btree indexes.
  • Corey Huinker sent in another revision of a patch to allow FETCH limited by bytes.
  • Anastasia Lubennikova and Heikki Linnakangas traded patches to add index-only scans for GiST.
  • Michael Paquier sent in a patch to rm static libraries before rebuilding.
  • Stephen Frost sent in another revision of a patch to add role attributes for various operations.
  • Tom Lane sent in a patch to cache domain constraints properly.
  • Michael Paquier sent in a patch to move freeze parameters of VacuumStmt into a separate structure.
  • Andrew Dunstan sent in another revision of a patch to implement mogrify and indent features for jsonb.

par N Bougain le jeudi 5 mars 2015 à 09h32

lundi 23 février 2015

Damien Clochard

PostgreSQL Dashboard

Un écran de suivi temps-réel basé sur Dashing et Sinatra

PostgreSQL Dashboard est outil de supervision très simple qui donne une vue instantanée de la santé d’une instance PostgreSQL.

L’outil est conçu pour être affiché sur un grand écran dans un salle de contrôle ou dans un espace de travail partagé.

Le tableau de bord actuel est composé de 5 “widgets” :

  • General Info : Version, nombre de bases, etc.
  • Hit Ratio : Le % de données trouvées en cache
  • Buffers : Le nombre de nouveaux buffers alloués
  • Queries : Le nombre de requête en cours sur l’instance
  • Twitter : Un aperçu du hashtag #PostgreSQL

Extensibilité

Ajouter un widget supplémentaire est très simple : cet outil est conçu pour faciliter l’écriture de widget “maison” et afficher des indicateurs métier.

Par ailleurs, la mise en page est totalement flexible. Vous pouvez simplement faire un drag’n’drop sur chaque widget et le placer à l’endroit qui vous convient. Le code HTML du tableau de bord peut être modifié pour des besoins spécifiques comme par exemple les dimensions d’un écran.

Liens & Remerciements

Certaines parties de PostgreSQL Dashboard sont basées sur le travail effectués pour d’autres outil PostgreSQL, notamment pgstats et pgcluu

PostgreSQL Dashboard est un projet ouvert distribué sous licence PostgreSQL. Toute contribution constructive est la bienvenue ! Vous pouvez envoyer vos idées, vos demandes et vos patches via les outils de GitHub.

Pour aller plus loin :

lundi 23 février 2015 à 21h27

dimanche 22 février 2015

Guillaume Lelarge

Durée d'établissement d'une connexion

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

J'ai toujours eu en tête qu'une connexion mettait du temps à s'établir entre un client et PostgreSQL. J'avais en tête un nombre qui me semblait plausible mais j'avoue que je n'avais jamais fait réellement le test.

Ce week-end, travaillant sur le chapitre sur la gestion des connexions, je me suis demandé si on pouvait calculer ce temps. J'ai donc regardé le code des processus postmaster/postgres pour ajouter quelques traces, histoire d'en savoir plus. Voici le patch que j'ai réalisé :

diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index f05114d..9d8fb8a 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2198,6 +2198,8 @@ ConnCreate(int serverFd)
 {
        Port       *port;
 
+       elog(LOG, "patch - ConnCreate(%d)", serverFd);
+
        if (!(port = (Port *) calloc(1, sizeof(Port))))
        {
                ereport(LOG,
@@ -3760,6 +3762,8 @@ BackendStartup(Port *port)
        Backend    *bn;                         /* for backend cleanup */
        pid_t           pid;
 
+       elog(LOG, "patch - BackendStart()");
+
        /*
         * Create backend data structure.  Better before the fork() so we can
         * handle failure cleanly.
@@ -3814,6 +3818,8 @@ BackendStartup(Port *port)
 
                MyProcPid = getpid();   /* reset MyProcPid */
 
+               elog(LOG, "patch - new pid is %d", MyProcPid);
+
                MyStartTime = time(NULL);
 
                /* We don't want the postmaster's proc_exit() handlers */
@@ -3916,6 +3922,8 @@ BackendInitialize(Port *port)
        char            remote_port[NI_MAXSERV];
        char            remote_ps_data[NI_MAXHOST];
 
+       elog(LOG, "patch - BackendInitialize()");
+
        /* Save port etc. for ps status */
        MyProcPort = port;
 
@@ -4096,6 +4104,8 @@ BackendRun(Port *port)
        int                     usecs;
        int                     i;
 
+       elog(LOG, "patch - BackendRun()");
+
        /*
         * Don't want backend to be able to see the postmaster random number
         * generator state.  We have to clobber the static random_seed *and* start
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index bc4eb33..4e1a3f7 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3578,6 +3578,7 @@ PostgresMain(int argc, char *argv[],
        sigjmp_buf      local_sigjmp_buf;
        volatile bool send_ready_for_query = true;
 
+       elog(LOG, "patch - PostgresMain()");
        /*
         * Initialize globals (already done if under postmaster, but not if
         * standalone).
@@ -3845,6 +3846,8 @@ PostgresMain(int argc, char *argv[],
         * were inside a transaction.
         */
 
+       elog(LOG, "patch - PostgresMain() - ready to execute command");
+
        if (sigsetjmp(local_sigjmp_buf, 1) != 0)
        {
                /*
@@ -4056,12 +4059,16 @@ PostgresMain(int argc, char *argv[],
                if (ignore_till_sync && firstchar != EOF)
                        continue;
 
+               elog(LOG, "patch - PostgresMain() - processing command");
+
                switch (firstchar)
                {
                        case 'Q':                       /* simple query */
                                {
                                        const char *query_string;
 
+                                       elog(LOG, "patch - PostgresMain() - executing simple query");
+
                                        /* Set statement_timestamp() */
                                        SetCurrentStatementStartTimestamp();
 
@@ -4279,6 +4286,8 @@ PostgresMain(int argc, char *argv[],
                        case 'X':
                        case EOF:
 
+                               elog(LOG, "patch - PostgresMain() - exiting");
+
                                /*
                                 * Reset whereToSendOutput to prevent ereport from attempting
                                 * to send any more messages to client.

En configurant PostgreSQL pour qu'il ajoute la date (à la milliseconde près) et le PID, et en configurant la trace des connexions et déconnexions :

log_min_duration_statement = 0
log_connections = on
log_disconnections = on
log_line_prefix = '%m [%p] '

et en exécutant la commande suivante :

$ psql -c "select * from t1 limit 200" b1

nous obtenons les traces suivantes :

2015-02-22 22:47:23.022 CET [6087] LOG:  patch - ConnCreate(5)
2015-02-22 22:47:23.022 CET [6087] LOG:  patch - BackendStart()
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - new pid is 6283
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - BackendInitialize()
2015-02-22 22:47:23.023 CET [6283] LOG:  connection received: host=[local]
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - BackendRun()
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - PostgresMain()
2015-02-22 22:47:23.025 CET [6283] LOG:  connection authorized: user=postgres database=b1
2015-02-22 22:47:23.027 CET [6283] LOG:  patch - PostgresMain() - ready to execute command
2015-02-22 22:47:23.027 CET [6283] LOG:  patch - PostgresMain() - processing command
2015-02-22 22:47:23.028 CET [6283] LOG:  patch - PostgresMain() - executing simple query
2015-02-22 22:47:23.028 CET [6283] LOG:  duration: 0.691 ms  statement: select * from t1 limit 200
2015-02-22 22:47:23.736 CET [6283] LOG:  patch - PostgresMain() - processing command
2015-02-22 22:47:23.736 CET [6283] LOG:  patch - PostgresMain() - exiting
2015-02-22 22:47:23.737 CET [6283] LOG:  disconnection: session time: 0:00:00.913 user=postgres database=b1 host=[local]

Autrement dit, il faut compter quelques millisecondes pour établir une connexion sans pooler. Après différents tests (impliquant notamment pgbench), le pire que j'ai vu est 10 millisecondes. Pas bien méchant quand on y pense. J'ai aussi noté que la toute première connexion était bien plus lente (dans les 40 millisecondes), ce qui reste encore bien loin de ce que j'imaginais.

J'ai aussi testé avec différentes valeurs du shared_buffers car il semblerait que la taille mémoire d'un processus a une importance dans la durée d'exécution de l'appel système fork().

Comme quoi il est vraiment préférable de tout tester pour ne pas avoir d'idées préconçues.

par Guillaume Lelarge le dimanche 22 février 2015 à 21h49

jeudi 19 février 2015

Guillaume Lelarge

PostgreSQL et la mémoire partagée

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

Lors de l'écriture du chapitre sur la gestion de la mémoire par PostgreSQL, j'ai cherché à différencier les différents blocs alloués en mémoire partagée. La documentation de PostgreSQL est assez peu bavard sur ce sujet, je me suis donc retourné vers le code source. Ce dernier donne énormément d'informations à qui se donne un peu la peine de les chercher. J'ai fini par trouver le fichier src/backend/storage/ipc/shmem.c qui s'occupe de la gestion de la mémoire partagée (shmem pour SHared MEMory).

Ce fichier contient différentes fonctions, dont la fonction ShmemInitStruct, dont le but est d'initialiser une structure en mémoire partagée. Il suffit de lui fournir le nom de la structure et sa taille, et la fonction se charge de l'allocation. Une petite modification de cette fonction permet d'afficher quelques informations à chaque appel de cette fonction. Voici ce patch :

diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 2ea2216..8703f48 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -402,6 +402,8 @@ ShmemInitStruct(const char *name, Size size, bool *foundPtr)
 	{
 		/* It isn't in the table yet. allocate and initialize it */
 		structPtr = ShmemAlloc(size);
+		ereport(LOG, (errmsg("allocate shared memory segment "%s", size %d",
+				name, (int)size)));
 		if (structPtr == NULL)
 		{
 			/* out of memory; remove the failed ShmemIndex entry */

Au lancement de PostgreSQL, modifié avec ce patch, nous obtenons cette sortie dans les traces :

LOG:  allocate shared memory segment "Control File", size 248 
LOG:  allocate shared memory segment "XLOG Ctl", size 16804496
LOG:  allocate shared memory segment "CLOG Ctl", size 525312
LOG:  allocate shared memory segment "SUBTRANS Ctl", size 263168
LOG:  allocate shared memory segment "MultiXactOffset Ctl", size 65856
LOG:  allocate shared memory segment "MultiXactMember Ctl", size 131648
LOG:  allocate shared memory segment "Shared MultiXact State", size 176 
LOG:  allocate shared memory segment "Buffer Descriptors", size 33554432
LOG:  allocate shared memory segment "Buffer Blocks", size 0
LOG:  allocate shared memory segment "Shared Buffer Lookup Table", size 32880
LOG:  allocate shared memory segment "Buffer Strategy Status", size 32
LOG:  allocate shared memory segment "LOCK hash", size 2160
LOG:  allocate shared memory segment "PROCLOCK hash", size 2160
LOG:  allocate shared memory segment "Fast Path Strong Relation Lock Data", size 4100
LOG:  allocate shared memory segment "PREDICATELOCKTARGET hash", size 2160
LOG:  allocate shared memory segment "PREDICATELOCK hash", size 2160
LOG:  allocate shared memory segment "PredXactList", size 88
LOG:  allocate shared memory segment "SERIALIZABLEXID hash", size 2160
LOG:  allocate shared memory segment "RWConflictPool", size 24
LOG:  allocate shared memory segment "FinishedSerializableTransactions", size 16
LOG:  allocate shared memory segment "OldSerXid SLRU Ctl", size 131648
LOG:  allocate shared memory segment "OldSerXidControlData", size 16
LOG:  allocate shared memory segment "Proc Header", size 88
LOG:  allocate shared memory segment "Proc Array", size 108 
LOG:  allocate shared memory segment "Backend Status Array", size 3672
LOG:  allocate shared memory segment "Backend Application Name Buffer", size 1088
LOG:  allocate shared memory segment "Backend Client Host Name Buffer", size 1088
LOG:  allocate shared memory segment "Backend Activity Buffer", size 17408
LOG:  allocate shared memory segment "Prepared Transaction Table", size 16
LOG:  allocate shared memory segment "Background Worker Data", size 1992
LOG:  allocate shared memory segment "shmInvalBuffer", size 66104
LOG:  allocate shared memory segment "PMSignalState", size 180 
LOG:  allocate shared memory segment "ProcSignalSlots", size 864 
LOG:  allocate shared memory segment "Checkpointer Data", size 10485800
LOG:  allocate shared memory segment "AutoVacuum Data", size 224 
LOG:  allocate shared memory segment "Wal Sender Ctl", size 56
LOG:  allocate shared memory segment "Wal Receiver Ctl", size 1192
LOG:  allocate shared memory segment "BTree Vacuum State", size 216 
LOG:  allocate shared memory segment "Sync Scan Locations List", size 656 
LOG:  allocate shared memory segment "Async Queue Control", size 244 
LOG:  allocate shared memory segment "Async Ctl", size 65856
LOG:  allocate shared memory segment "pg_stat_statements", size 48
LOG:  allocate shared memory segment "pg_stat_statements hash", size 2160

(Et comme on peut le constater par les deux dernières lignes, ce serveur avait l'extension pg_stat_statements à charger)

Nous connaissons ainsi les différents segments et leur taille respective. À partir de là, il a suffit de suivre la piste des différents segments pour comprendre leur utilité et la façon dont ils sont utilisés.

par Guillaume Lelarge le jeudi 19 février 2015 à 22h53

mercredi 21 janvier 2015

Guillaume Lelarge

PostgreSQL - Architecture et notions avancées

Et voilà, j'ai fini par le faire. J'ai fini par me convaincre d'écrire un livre complet sur PostgreSQL.

C'est quelque chose qui me trottait dans la tête depuis longtemps. Surtout depuis que Thomas Reiss m'avait montré le Concept Guide d'Oracle. Je m'étais dit à ce moment-là : mince, c'est ça qu'il manque à PostgreSQL. Mais bon, ça demande beaucoup de temps, beaucoup de travail. Je n'avais pas très envie de me jeter là-dedans, même si je savais que certains éditeurs cherchaient des auteurs pour des livres sur PostgreSQL.

J'ai eu la chance de connaître Patricia Montcorgé lors de la traduction du livre de Greg Smith sur les performances avec PostgreSQL. Si bien que, après avoir lu le livre de Brendan Gregg sur la recherche de performances (Systems Performance: Enterprise and the Cloud, excellent livre, à mettre entre toutes les mains), je lui ai proposé deux projets : la traduction de ce livre et l'écriture d'un livre sur PostgreSQL. Elle m'a appris qu'elle avait fondé sa propre maison d'édition, qu'elle ne s'occupait plus de traductions, et que le livre sur PostgreSQL pourrait l'intéresser. On s'est rencontré, et, après lui avoir expliqué plus en profondeur mon projet, elle a pu m'expliquer comment elle voyait le travail avec elle. J'ai trouvé que ça ressemblait beaucoup à un projet libre, avec une version beta, du git, du docbook, des mises à jour facile, etc. Bref, j'étais bien accroché.

Je travaille sur ce livre depuis avril 2014. On a déjà subi un gros retard à cause du chapitre sur les processus : étudier le code de chaque processus a été particulièrement long. Mais bon, on a maintenant un rythme de croisière intéressant. On s'est décidé sur un chapitre tous les 4 à 6 semaines.

La version beta est sortie aujourd'hui. Elle est disponible par module ou complète, uniquement en version électronique pour le moment (jusqu'à la version finale en fait). Les chapitres seront livrés au fur et à mesure de leur écriture. Un forum est disponible pour les lecteurs qui voudraient laisser des commentaires ou des demandes ou des corrections.

La version finale sera mise à jour pour intégrer les changements effectués par la version 9.5.

Bon, je retourne bosser sur le livre :)

Preneur de tout commentaire par mail sur guillaume@lelarge.info.

par Guillaume Lelarge le mercredi 21 janvier 2015 à 22h50

dimanche 28 décembre 2014

Guillaume Lelarge

Manuel de la 9.4 et articles GLMF

Trois billets en 2014 (avec celui-là), j'aurais fait fort :)

Néanmoins, quelques informations importantes.

Tout d'abord, la documentation de PostgreSQL 9.4 est enfin disponible entièrement en français, avec les différents formats habituels. C'est comme d'habitude sur http://docs.postgresql.fr.

Ensuite, au niveau des articles pour le GNU/Linux Magazine France, deux articles sont parus sur la version 9.4, un peu avant la sortie (les numéros 175 et 177). La suite de la série sur le planificateur est prévue rapidement avec un article sur les jointures. Il s'agit donc du deuxième article. Les troisième et quatrième articles sont déjà dans le bureau du rédacteur en chef (depuis août en fait). Ils devraient sortir rapidement après, suivant la place disponible dans le magazine.

Et voilà. L'année 2015 devrait être très intéressante :) Quelques infos plus tard à ce sujet.

par Guillaume Lelarge le dimanche 28 décembre 2014 à 11h36

lundi 22 septembre 2014

Nicolas Thauvin

Confiner PostgreSQL avec SELinux

J'avais déjà expérimenté un peu avec SELinux il y a deux ans sans aller trop loin, parce que j'entendais souvent la phrase "Si on veut de la sécurité, il faut SELinux" et surtout à cause de l'arrivée de l'extension sepgsql dans les modules contrib de PostgreSQL. Ça avait donné une conf pour le Fosdem où finalement, j'ai plus parlé des privilèges classiques que de SELinux.

Suite à une demande au ${BOULOT}, je me suis replongé dedans, et les choses ont peu évolué. Dans la majorité des recherches que j'ai pu faire, SELinux reste tout de même un truc qui se met en travers la route, c'est-à-dire, qu'il y a toujours plus de résultats sur comment le désactiver plutôt que configurer les choses correctement. Ensuite, certains proposent de faire aveuglement confiance à setroubleshoot et audit2allow pour faire taire la bête. Enfin, j'ai dû trouver une page ou deux après en deux semaines à temps plein sur le sujet qui parlent de comment confiner un utilisateur dans le but d'implémenter ce que promet SELinux et que toutes les entreprises demandent : le RBAC, Role Based Access Control, ou comment donner le moins de droits possibles aux sous-traitants qui hébergent les serveurs.

Cette fois ci, je suis allé plus loin. Déjà, je suis parti sur une installation de CentOS 6, la famille de distribution RHEL/CentOS/Fedora semble être la plus en avance par rapport à SELinux : à peu près tous les services/daemons fournis dans l'install sont confinés par SELinux. Les utilisateurs ne le sont pas au login, il n'y a donc aucune configuration de RBAC. On verra peut-être ça plus tard, j'ai pas encore obtenu de résultat satisfaisant sur ce sujet, et il vaut mieux expliquer comment confiner correctement PostgreSQL avant de passer aux roles. Tout simplement, parce qu'il est très simple de casser ce confinement avec la configuration par défaut de RHEL, grâce à pg_ctl. Enfin, l'installation des paquets du PGDG n'est pas confinée par défaut, il manque les file contexts adaptés aux chemins particuliers de ces paquets, prévu pour faire cohabiter plusieurs versions majeures ; ce qui ne correspond pas ce qu'à prévu Red Hat pour PostgreSQL.

Après cette longue introduction, mais avant de commencer, il faut savoir administrer un minumum SELinux : si vous ne savez pas qu'il existe des options -Z, ce que sont les contexts, les types et les domaines, mieux vaut d'abord se documenter, par exemple chez Red Hat.

Voici donc les file contexts à ajouter dans un fichier module.te pour confiner l'installation PostgreSQL du PGDG :

/etc/rc\.d/init\.d/(se)?postgresql(-.*)?    --  gen_context(system_u:object_r:postgresql_initrc_exec_t,s0)
/usr/pgsql-[0-9]+\.[0-9]+/bin/initdb        --  gen_context(system_u:object_r:postgresql_exec_t,s0)
/usr/pgsql-[0-9]+\.[0-9]+/bin/pg_ctl        --  gen_context(system_u:object_r:postgresql_initrc_exec_t,s0)
/usr/pgsql-[0-9]+\.[0-9]+/bin/postgres      --  gen_context(system_u:object_r:postgresql_exec_t,s0)
/usr/pgsql-[0-9]+.[0-9]+/share/locale(/.*)?     gen_context(system_u:object_r:locale_t,s0)
/usr/pgsql-[0-9]+.[0-9]+/share/man(/.*)?        gen_context(system_u:object_r:man_t,s0)

Tout d'abord, pour l'init script et pg_ctl, on utilise le type postgresql_initrc_exec_t, c'est ce qui permet de lancer PostgreSQL dans le domaine confiné postgresql_t au boot, via l'init script, et manuellement. La méthode la plus propre est de toujours utiliser l'init script, idéalement par l'intermédiaire de run_init pour démarrer, arrêter ou redémarrer le postmaster. On évite alors de laisser trainer des choses dans /var/{run,lock}.

Les programmes postgres et initdb doivent avoir le type postgresql_exec_t car ils exécutent le serveur PostgreSQL ; cela doit se faire dans le domaine postgresql_t.

Enfin, on a placé les labels corrects sur les fichiers de traduction et les pages de man, pour faire plus propre. Ce code source de module SELinux alors doit être compilé et chargé.

Cette configuration est reprise dans le module SELinux disponible sur github. On peut aussi l'ajouter manuellement avec semanage :

semanage fcontext -a -t postgresql_initrc_exec_t '/etc/rc\.d/init\.d/(se)?postgresql(-.*)?'
semanage fcontext -a -t postgresql_exec_t '/usr/pgsql-[0-9]+\.[0-9]+/bin/initdb'
semanage fcontext -a -t postgresql_initrc_exec_t '/usr/pgsql-[0-9]+\.[0-9]+/bin/pg_ctl'
semanage fcontext -a -t postgresql_exec_t '/usr/pgsql-[0-9]+\.[0-9]+/bin/postgres'
semanage fcontext -a -t locale_t '/usr/pgsql-[0-9]+.[0-9]+/share/locale(/.*)?'
semanage fcontext -a -t man_t '/usr/pgsql-[0-9]+.[0-9]+/share/man(/.*)?'

Il existe un certain nombre de booleans pour la configuration des droits SELinux de PostgreSQL, le plus important concerne rsync, souvent nécessaire pour faire des base backups. Il s'agit de postgresql_can_rsync, pour l'activer :

semanage boolean -m --on postgresql_can_rsync

Si on lance l'instance sur un port TCP différent de 5432, il faut l'autoriser dans la configuration locale de SELinux :

semanage port -a -t postgresql_port_t -p tcp <port>

Enfin, il ne faut pas oublier d'appliquer les contexts aux fichiers soit avec restorecon, un relabel complet au reboot ou chcon.

lundi 22 septembre 2014 à 14h23

Rodolphe Quiédeville

Django Paginator oui mais

Après la lecture de l'excellent article de Markus Winnand no-offset j'ai réalisé un test pour mesurer la différence de performance de l'utilisation d'OFFSET avec Django.

La suite à lire sur le blog Novapost's paradize.

par Rodolphe Quiédeville le lundi 22 septembre 2014 à 08h13

samedi 12 avril 2014

Guillaume Lelarge

Nouvelle série d'articles pour GLMF

Après l'article sur les nouveautés de la version 9.3, j'ai entamé une série d'articles sur le fonctionnement du planificateur (aussi appelé optimiseur).

Le premier est sorti ce mois-ci, dans le numéro 170 de GNU/Linux Magazine France. Il traite des différents types de parcours que l'optimiseur peut planifier.

J'ai loupé le coche pour le prochain numéro, donc celui sur les jointures devrait sortir dans le numéro 172 si tout va bien. En fait, je viens tout juste de finir son écriture.

Il y aura certainement deux autres articles, un sur les autres types de nœuds et un sur les outils pour la compréhension des plans d'exécution mais ils restent à écrire.

En attendant, je suis preneur de toute remarque/critique sur mes articles :)

par Guillaume Lelarge le samedi 12 avril 2014 à 16h11

mardi 11 mars 2014

Rodolphe Quiédeville

Debian, PG9.3, Osmosis et Nominatim

Même en se basant sur des références en terme de stabilité il peut arriver que certains combos soient fatals à votre production. C'est ce qui m'est arrivé récemment pour un serveur Nominatim installé pourtant sur une Debian Wheezy, osmosis est utilisé pour la mise à jour continue de la base Nominatim et m'a fait des misères que je m'en vais vous conter.

En parallèle de la version de PostgreSQL 9.1 standard Debian j'ai installé sur la machine une 9.3 en provenance du dépôt Debian de la communauté PG (le support dfe JSON dans PG 9.3 c'est awesome), jusque là tout va bien. Seulement à l'upgrade suivant le paquet libpostgis-java est passé en version 2.1.1-5.pgdg70+1 (celle-ci étant disponible sur le dépôt PG) ; malheureusement cette version est incompatible avec osmosis packagé chez Debian qui nécessite la version 1.5.3 de cette librairie, et là c'est le drâme !

Donc si au lancement d'osmosis vous rencontrez l'erreur :

java.io.FileNotFoundException: /usr/share/java/postgis.jar

Alors que le sus-dit fichier est bien présent, et que vous commencez à dire du mal des backtraces java, il existe un solution simple et efficace, downgrader la version de libpostgis-java en quelques commandes :

dpkg --remove osmosis
apt-get install libpostgis-java=1.5.3-2 osmosis

Sans oublier un pinning pour éviter la future mise à jour du paquet.

par Rodolphe Quiédeville le mardi 11 mars 2014 à 08h07