From nobody Fri Apr 19 20:58:36 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1631652312; cv=none; d=zohomail.com; s=zohoarc; b=Xki4TENdhiliL2WlLY5sXHuZ0AIuKU+3HNpVbrtnTRJTnhSDtPhP5er5aYFQDifzud1Qg3AgpHD+/1cqwLwhOOHicob/yC8gM1oYKgkCPr/amIlkGEKHS/ijpV4q8KevVDl/UiYdVP+vAwPGoYwWzUwdIUqv6kcadZvWGXCT2Tw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1631652312; h=Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:Message-ID:Sender:Subject:To; bh=GoZcnLD9m4N5gmlIbm/gTIsyYg9LzB8Uf3nV7DR5ekM=; b=Sy4QRbeceNZwoTRXb1ZzyPN/DGVghyujcDPTXFhRebyFI7CAYDIlIoyTUXjD18RSf5/35Nda7m6N5IeVHuUI8loUlcm+747CGYYGctDlg5O8wgXIbG2lkM4aypYpmH7UUbODyitOIQj+Piy8j+N+YpQe8YLB1c2T8A/Ascwm8yE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1631652312393269.3300455374399; Tue, 14 Sep 2021 13:45:12 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.187010.335741 (Exim 4.92) (envelope-from ) id 1mQFIF-00068S-HB; Tue, 14 Sep 2021 20:44:39 +0000 Received: by outflank-mailman (output) from mailman id 187010.335741; Tue, 14 Sep 2021 20:44:39 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mQFIF-00068L-E7; Tue, 14 Sep 2021 20:44:39 +0000 Received: by outflank-mailman (input) for mailman id 187010; Tue, 14 Sep 2021 20:44:38 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mQFIE-00068F-63 for xen-devel@lists.xenproject.org; Tue, 14 Sep 2021 20:44:38 +0000 Received: from mail-wr1-x42a.google.com (unknown [2a00:1450:4864:20::42a]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 21744033-2fad-4d17-8d18-c364a8c09127; Tue, 14 Sep 2021 20:44:36 +0000 (UTC) Received: by mail-wr1-x42a.google.com with SMTP id x6so257295wrv.13 for ; Tue, 14 Sep 2021 13:44:36 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id k6sm993474wmo.37.2021.09.14.13.44.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 Sep 2021 13:44:34 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 21744033-2fad-4d17-8d18-c364a8c09127 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id; bh=GoZcnLD9m4N5gmlIbm/gTIsyYg9LzB8Uf3nV7DR5ekM=; b=ZNjP0eQIp2w0o0AZAfIaJg4kkVe8XY4Ywq6flQ5pJQtgwGd0l60WMm1TXeWMEXZ3dH tb1jWTijtIBo7CnaHoEIeFEEx2gsDxjlbApoe2e31Os1q7tLN7r1yyhvE5/sZPeJUMBS UrxO0wfKOTEMEMvJKNI0CB2lR9GZAZdfgAbSxaG+q4F+Ln+/B6gig9507i/0B/O97cdJ aLrqUmwp5Yu3SLYdit5AKzvVfBXcRCBj5YQ8YzQfsuv2/+95Up6jtGuZ+f15RD+8hINh Hv4+46IgYQSFsiKjDp9vAcAIXPSBWPVcP1WQAIfOxg5x++f5aMa3gSsHU2MrWaUoTlQh g/Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=GoZcnLD9m4N5gmlIbm/gTIsyYg9LzB8Uf3nV7DR5ekM=; b=lxEGncDmmnD412WJrUOgOr5p9uLiY9yxEImiGLJPw6C2iUCpwHQmb9O6vySzV4puoO oU6EZwfOrM4eZ94CMJsF0vcMNSk8d80GTKjWAQI11SH6xUnkaSjEeell/2xVXpeHp+ch vwQi1IKzev6IcHYkZdTD1vDLRZrs1YMwwsv/rbCQylb9H4e4/lqIo5R0FAtxQxODHohs 4INahhn0mj3KMzx6I92rA8vVwKJ4gEr9pP+YWZDn/R0bj0U0i4XNVou6HpPyPbEuTXtv U63HN3lSuDhsYj3cnzRc73CiBjtstgJ7B6A6F4Zy6RWAUOeLArscaCtiP5TysBXop/ko j+Aw== X-Gm-Message-State: AOAM531LiU7CBYynJACtRiXnSvEOoyrelp9uhTL/zFROCVFIDIAyamkd tJUcOefe+hhlQMBUmgYYFvSIeWzxEc0= X-Google-Smtp-Source: ABdhPJxvkGIyo//M06CXfKTh9hza52v2LJrPQ4HUdo3pNaZeSW1u9gsSsHN2XCmd62H7kggu4ctS5w== X-Received: by 2002:adf:c501:: with SMTP id q1mr1160367wrf.150.1631652274788; Tue, 14 Sep 2021 13:44:34 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Tyshchenko , Stefano Stabellini , Julien Grall , Volodymyr Babchuk , Andrew Cooper , George Dunlap , Ian Jackson , Jan Beulich , Wei Liu , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Subject: [RFC PATCH V2] xen/gnttab: Store frame GFN in struct page_info on Arm Date: Tue, 14 Sep 2021 23:44:05 +0300 Message-Id: <1631652245-30746-1-git-send-email-olekstysh@gmail.com> X-Mailer: git-send-email 2.7.4 X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1631652315079100001 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Oleksandr Tyshchenko Rework Arm implementation to store grant table frame GFN in struct page_info directly instead of keeping it in standalone status/shared arrays. To cover 64-bit/40-bit IPA on Arm64/Arm32 we need the space to hold 52-bit/28-bit value respectively. In order to not grow the size of struct page_info borrow the required amount of bits from type_info's count portion which current context won't suffer (currently only 1 bit is used on Arm). Introduce corresponding PGT_* constructs and access macros. Update existing gnttab macros to deal with GFN value according to new location. Also update the use of count portion on Arm in share_xen_page_with_guest(). Update the P2M code to clean said GFN portion when putting a reference on the grant table page in p2m_put_l3_page(). The added check is based on the assumption that grant table page is the xen_heap page and its entry has p2m_ram_rw type, which is correct. However, this check is not entirely precise and we might end up clearing the GFN portion for other xen_heap pages with the same p2m_type. But, this action is considered as harmless, since only grant table pages really use that portion. And for everything to work correctly introduce arch-specific macros to be called from alloc_xenheap_pages()/free_xenheap_pages() which purposes on Arm are to clear the GFN portion before use and make sure the portion is cleared after use, on x86 these are just stubs. This patch is intended to fix the potential issue on Arm which might happen when remapping grant-table frame. A guest (or the toolstack) will unmap the grant-table frame using XENMEM_remove_physmap. This is a generic hypercall, so on x86, we are relying on the fact the M2P entry will be cleared on removal. For architecture without the M2P, the GFN would still be present in the grant frame/status array. So on the next call to map the page, we will end up to request the P2M to remove whatever mapping was the given GFN. This could well be another mapping. Besides that, this patch simplifies arch code on Arm by removing arrays and corresponding management code and as the result gnttab_init_arch/gnttab_destroy_arch helpers and struct grant_table_arch become useless and can be dropped globally. Suggested-by: Julien Grall Signed-off-by: Oleksandr Tyshchenko --- You can find the related discussions at: https://lore.kernel.org/xen-devel/93d0df14-2c8a-c2e3-8c51-54412190171c@xen.= org/ https://lore.kernel.org/xen-devel/1628890077-12545-1-git-send-email-oleksty= sh@gmail.com/ https://lore.kernel.org/xen-devel/1631228688-30347-1-git-send-email-oleksty= sh@gmail.com/ ! Please note, there is still unresolved locking question here for which I failed to find a suitable solution ! According to the internal conversation: Now the GFN field in the struct page_info is accessed from gnttab_set_frame_gfn() in the grant table code and from page_set_frame_gfn() in the P2M code (the former uses the latter). We need to prevent the concurrent access to this field. But, we cannot grab the grant lock from the P2M code because we will introduce a lock inversion. The page_set_frame_gfn() will be called from the P2M code with the p2m lock= held and then acquire the grant table lock. The gnttab_map_frame() will do the i= nverse. Changes RFC1 -> RFC2: - update patch description - add/update comments in code - clarify check in p2m_put_l3_page() - introduce arch_alloc_xenheap_page() and arch_free_xenheap_page() and drop page_arch_init() - add ASSERT to gnttab_shared_page() and gnttab_status_page() - rework changes to Arm's struct page_info: do not split type_info, allocate GFN portion by reducing count portion, create corresponding PGT_* construct, etc - update page_get_frame_gfn() and page_set_frame_gfn() - update the use of count portion on Arm - drop the leading underscore in the macro parameter names --- xen/arch/arm/mm.c | 8 ++++-- xen/arch/arm/p2m.c | 21 ++++++++++++--- xen/common/grant_table.c | 9 ------- xen/common/page_alloc.c | 6 +++++ xen/include/asm-arm/grant_table.h | 57 +++++++++++++++--------------------= ---- xen/include/asm-arm/mm.h | 36 ++++++++++++++++++++++--- xen/include/asm-x86/grant_table.h | 5 ---- xen/include/asm-x86/mm.h | 4 +++ 8 files changed, 87 insertions(+), 59 deletions(-) diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index eea926d..1857af4 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -1376,14 +1376,18 @@ unsigned long domain_get_maximum_gpfn(struct domain= *d) void share_xen_page_with_guest(struct page_info *page, struct domain *d, enum XENSHARE_flags flags) { + unsigned long type_info; + if ( page_get_owner(page) =3D=3D d ) return; =20 spin_lock(&d->page_alloc_lock); =20 /* The incremented type count pins as writable or read-only. */ - page->u.inuse.type_info =3D - (flags =3D=3D SHARE_ro ? PGT_none : PGT_writable_page) | 1; + type_info =3D page->u.inuse.type_info & ~(PGT_type_mask | PGT_count_ma= sk); + page->u.inuse.type_info =3D type_info | + (flags =3D=3D SHARE_ro ? PGT_none : PGT_writable_page) | + (1UL << PGT_count_base); =20 page_set_owner(page, d); smp_wmb(); /* install valid domain ptr before updating refcnt. */ diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c index eff9a10..a2b5597 100644 --- a/xen/arch/arm/p2m.c +++ b/xen/arch/arm/p2m.c @@ -718,8 +718,10 @@ static int p2m_mem_access_radix_set(struct p2m_domain = *p2m, gfn_t gfn, * TODO: Handle superpages, for now we only take special references for le= af * pages (specifically foreign ones, which can't be super mapped today). */ -static void p2m_put_l3_page(const lpae_t pte) +static void p2m_put_l3_page(struct p2m_domain *p2m, const lpae_t pte) { + mfn_t mfn =3D lpae_get_mfn(pte); + ASSERT(p2m_is_valid(pte)); =20 /* @@ -731,11 +733,22 @@ static void p2m_put_l3_page(const lpae_t pte) */ if ( p2m_is_foreign(pte.p2m.type) ) { - mfn_t mfn =3D lpae_get_mfn(pte); - ASSERT(mfn_valid(mfn)); put_page(mfn_to_page(mfn)); } + +#ifdef CONFIG_GRANT_TABLE + /* + * Check whether we deal with grant table page. As the grant table page + * is xen_heap page and its entry has known p2m type, detect it and ma= rk + * the stored GFN as invalid. Although this check is not precise and we + * might end up updating this for other xen_heap pages, this action is + * harmless to these pages since only grant table pages have this field + * in use. So, at worst, unnecessary action might be performed. + */ + if ( (pte.p2m.type =3D=3D p2m_ram_rw) && is_xen_heap_mfn(mfn) ) + page_set_frame_gfn(mfn_to_page(mfn), INVALID_GFN); +#endif } =20 /* Free lpae sub-tree behind an entry */ @@ -768,7 +781,7 @@ static void p2m_free_entry(struct p2m_domain *p2m, p2m->stats.mappings[level]--; /* Nothing to do if the entry is a super-page. */ if ( level =3D=3D 3 ) - p2m_put_l3_page(entry); + p2m_put_l3_page(p2m, entry); return; } =20 diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c index e80f8d0..fd8d7e3 100644 --- a/xen/common/grant_table.c +++ b/xen/common/grant_table.c @@ -93,8 +93,6 @@ struct grant_table { =20 /* Domain to which this struct grant_table belongs. */ const struct domain *domain; - - struct grant_table_arch arch; }; =20 unsigned int __read_mostly opt_max_grant_frames =3D 64; @@ -1981,14 +1979,9 @@ int grant_table_init(struct domain *d, int max_grant= _frames, =20 grant_write_lock(gt); =20 - ret =3D gnttab_init_arch(gt); - if ( ret ) - goto unlock; - /* gnttab_grow_table() allocates a min number of frames, so 0 is okay.= */ ret =3D gnttab_grow_table(d, 0); =20 - unlock: grant_write_unlock(gt); =20 out: @@ -3894,8 +3887,6 @@ grant_table_destroy( if ( t =3D=3D NULL ) return; =20 - gnttab_destroy_arch(t); - for ( i =3D 0; i < nr_grant_frames(t); i++ ) free_xenheap_page(t->shared_raw[i]); xfree(t->shared_raw); diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 958ba0c..00371e7 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -2204,7 +2204,10 @@ void *alloc_xenheap_pages(unsigned int order, unsign= ed int memflags) return NULL; =20 for ( i =3D 0; i < (1u << order); i++ ) + { pg[i].count_info |=3D PGC_xen_heap; + arch_alloc_xenheap_page(&pg[i]); + } =20 return page_to_virt(pg); } @@ -2222,7 +2225,10 @@ void free_xenheap_pages(void *v, unsigned int order) pg =3D virt_to_page(v); =20 for ( i =3D 0; i < (1u << order); i++ ) + { pg[i].count_info &=3D ~PGC_xen_heap; + arch_free_xenheap_page(&pg[i]); + } =20 free_heap_pages(pg, order, true); } diff --git a/xen/include/asm-arm/grant_table.h b/xen/include/asm-arm/grant_= table.h index 0ce77f9..479339d 100644 --- a/xen/include/asm-arm/grant_table.h +++ b/xen/include/asm-arm/grant_table.h @@ -11,11 +11,6 @@ #define INITIAL_NR_GRANT_FRAMES 1U #define GNTTAB_MAX_VERSION 1 =20 -struct grant_table_arch { - gfn_t *shared_gfn; - gfn_t *status_gfn; -}; - static inline void gnttab_clear_flags(struct domain *d, unsigned int mask, uint16_t *addr) { @@ -46,35 +41,11 @@ int replace_grant_host_mapping(unsigned long gpaddr, mf= n_t mfn, #define gnttab_dom0_frames() \ min_t(unsigned int, opt_max_grant_frames, PFN_DOWN(_etext - _stext)) =20 -#define gnttab_init_arch(gt) \ -({ \ - unsigned int ngf_ =3D (gt)->max_grant_frames; = \ - unsigned int nsf_ =3D grant_to_status_frames(ngf_); = \ - \ - (gt)->arch.shared_gfn =3D xmalloc_array(gfn_t, ngf_); = \ - (gt)->arch.status_gfn =3D xmalloc_array(gfn_t, nsf_); = \ - if ( (gt)->arch.shared_gfn && (gt)->arch.status_gfn ) \ - { \ - while ( ngf_-- ) \ - (gt)->arch.shared_gfn[ngf_] =3D INVALID_GFN; = \ - while ( nsf_-- ) \ - (gt)->arch.status_gfn[nsf_] =3D INVALID_GFN; = \ - } \ - else \ - gnttab_destroy_arch(gt); \ - (gt)->arch.shared_gfn ? 0 : -ENOMEM; \ -}) - -#define gnttab_destroy_arch(gt) \ - do { \ - XFREE((gt)->arch.shared_gfn); \ - XFREE((gt)->arch.status_gfn); \ - } while ( 0 ) - #define gnttab_set_frame_gfn(gt, st, idx, gfn) \ do { \ - ((st) ? (gt)->arch.status_gfn : (gt)->arch.shared_gfn)[idx] =3D = \ - (gfn); \ + struct page_info *pg_ =3D (st) ? gnttab_status_page(gt, idx) = \ + : gnttab_shared_page(gt, idx); \ + page_set_frame_gfn(pg_, gfn); \ } while ( 0 ) =20 #define gnttab_get_frame_gfn(gt, st, idx) ({ \ @@ -82,11 +53,25 @@ int replace_grant_host_mapping(unsigned long gpaddr, mf= n_t mfn, : gnttab_shared_gfn(NULL, gt, idx); \ }) =20 -#define gnttab_shared_gfn(d, t, i) \ - (((i) >=3D nr_grant_frames(t)) ? INVALID_GFN : (t)->arch.shared_gfn[i]) +#define gnttab_shared_page(t, i) ({ \ + ASSERT((t)->shared_raw[i]); \ + mfn_to_page(_mfn(__virt_to_mfn((t)->shared_raw[i]))); \ +}) + +#define gnttab_status_page(t, i) ({ \ + ASSERT((t)->status[i]); \ + mfn_to_page(_mfn(__virt_to_mfn((t)->status[i]))); \ +}) =20 -#define gnttab_status_gfn(d, t, i) \ - (((i) >=3D nr_status_frames(t)) ? INVALID_GFN : (t)->arch.status_gfn[i= ]) +#define gnttab_shared_gfn(d, t, i) ({ \ + struct page_info *pg_ =3D gnttab_shared_page(t, i); = \ + page_get_frame_gfn(pg_); \ +}) + +#define gnttab_status_gfn(d, t, i) ({ \ + struct page_info *pg_ =3D gnttab_status_page(t, i); = \ + page_get_frame_gfn(pg_); \ +}) =20 #define gnttab_need_iommu_mapping(d) \ (is_domain_direct_mapped(d) && is_iommu_enabled(d)) diff --git a/xen/include/asm-arm/mm.h b/xen/include/asm-arm/mm.h index ded74d2..dd425d6 100644 --- a/xen/include/asm-arm/mm.h +++ b/xen/include/asm-arm/mm.h @@ -98,9 +98,18 @@ struct page_info #define PGT_writable_page PG_mask(1, 1) /* has writable mappings? = */ #define PGT_type_mask PG_mask(1, 1) /* Bits 31 or 63. = */ =20 - /* Count of uses of this frame as its current type. */ -#define PGT_count_width PG_shift(2) -#define PGT_count_mask ((1UL<u.inuse.type_info & PGT_gfn_mask); \ + gfn_eq(gfn_, PGT_INVALID_FRAME_GFN) ? INVALID_GFN : gfn_; \ +}) + +#define page_set_frame_gfn(p, gfn) ({ \ + gfn_t gfn_ =3D gfn_eq(gfn, INVALID_GFN) ? \ + PGT_INVALID_FRAME_GFN : gfn; \ + (p)->u.inuse.type_info &=3D ~PGT_gfn_mask; \ + (p)->u.inuse.type_info |=3D gfn_x(gfn_); \ +}) + +/* + * As the struct page_info representing the xen_heap page can contain + * the grant table frame GFN on Arm we need to clear it beforehand and + * make sure it is not still set when freeing a page. + */ +#define arch_alloc_xenheap_page(p) page_set_frame_gfn(p, INVALID_GFN) +#define arch_free_xenheap_page(p) \ + BUG_ON(!gfn_eq(page_get_frame_gfn(p), INVALID_GFN)) + #define frame_table ((struct page_info *)FRAMETABLE_VIRT_START) /* PDX of the first page in the frame table. */ extern unsigned long frametable_base_pdx; diff --git a/xen/include/asm-x86/grant_table.h b/xen/include/asm-x86/grant_= table.h index 84e3296..0eb018f 100644 --- a/xen/include/asm-x86/grant_table.h +++ b/xen/include/asm-x86/grant_table.h @@ -14,9 +14,6 @@ =20 #define INITIAL_NR_GRANT_FRAMES 1U =20 -struct grant_table_arch { -}; - static inline int create_grant_host_mapping(uint64_t addr, mfn_t frame, unsigned int flags, unsigned int cache_flags) @@ -35,8 +32,6 @@ static inline int replace_grant_host_mapping(uint64_t add= r, mfn_t frame, return replace_grant_pv_mapping(addr, frame, new_addr, flags); } =20 -#define gnttab_init_arch(gt) 0 -#define gnttab_destroy_arch(gt) do {} while ( 0 ) #define gnttab_set_frame_gfn(gt, st, idx, gfn) do {} while ( 0 ) #define gnttab_get_frame_gfn(gt, st, idx) ({ \ mfn_t mfn_ =3D (st) ? gnttab_status_mfn(gt, idx) = \ diff --git a/xen/include/asm-x86/mm.h b/xen/include/asm-x86/mm.h index cb90527..04d8704 100644 --- a/xen/include/asm-x86/mm.h +++ b/xen/include/asm-x86/mm.h @@ -327,6 +327,10 @@ struct page_info =20 #define maddr_get_owner(ma) (page_get_owner(maddr_to_page((ma)))) =20 +/* No arch-specific actions are needed for the xen_heap page */ +#define arch_alloc_xenheap_page(p) do {} while ( 0 ) +#define arch_free_xenheap_page(p) do {} while ( 0 ) + #define frame_table ((struct page_info *)FRAMETABLE_VIRT_START) extern unsigned long max_page; extern unsigned long total_pages; --=20 2.7.4