袁建明, 赵成贵. 基于HDFS Federation的分布式局域网缓存系统设计[J]. 云南大学学报(自然科学版), 2014, 36(S2): 30-34. doi: 10.7540/j.ynu.2014a05
引用本文: 袁建明, 赵成贵. 基于HDFS Federation的分布式局域网缓存系统设计[J]. 云南大学学报(自然科学版), 2014, 36(S2): 30-34. doi: 10.7540/j.ynu.2014a05
YUAN Jianming, ZHAO Chenggui. The design of distributed cache system in LAN based on HDFS Federation[J]. Journal of Yunnan University: Natural Sciences Edition, 2014, 36(S2): 30-34. DOI: 10.7540/j.ynu.2014a05
Citation: YUAN Jianming, ZHAO Chenggui. The design of distributed cache system in LAN based on HDFS Federation[J]. Journal of Yunnan University: Natural Sciences Edition, 2014, 36(S2): 30-34. DOI: 10.7540/j.ynu.2014a05

基于HDFS Federation的分布式局域网缓存系统设计

The design of distributed cache system in LAN based on HDFS Federation

  • 摘要: 在大型局域网中部署内容缓存加速系统可以缓解网络出口的压力并提高用户的满意度.传统的方法是采用本地文件系统存储缓存文件,若分布式部署则导致冗余度高,造成资源的浪费;若单机部署则扩展性差,性能达不到要求.此系统可以将缓存文件存放在HDFS集群中,用户需要获取的文件若缓存中已经存在,则可以通过伪造重定向数据包让用户重定向到相应的本地服务器去获取文件.采用HDFS Federation可以部署多个NameNode来扩展集群的容量,降低单个NameNode的负载.实验结果表明在高负载下,本系统可以提供更高的传输速度.

     

    Abstract: Setting up a cache system in large LAN can relieve the pressure of WAN interface and increase user's satisfaction.The traditional system store cache files with the local file system.Deploying the traditional system distributed could lead to high redundancy,resulting in a waste of resources.And deploying the traditional system centralized could lead to poor scalability,resulting in lack of performance.The cache system presented in this paper can store the cache files into HDFS.If the file got by user exists in the system,the user could get a redirect packet containing the file path nearest to the user.The HDFS Federation can use multiple NameNodes to extend HDFS capacity and reduce the load of single NameNode.The experimental result shows that this system can provide faster speed under extreme load.

     

/

返回文章
返回